[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:35:42 dist-test-slave-06lh krb5kdc[6246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:35:42 dist-test-slave-06lh krb5kdc[6246](info): set up 2 sockets
Sep 28 14:35:42 dist-test-slave-06lh krb5kdc[6246](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 14:35:43.877071 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:35:43 dist-test-slave-06lh krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:35:44.002998 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39911
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:39911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:44.115460 6259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:44.115788 6259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:44.115867 6259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:44.119957 6259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:35:44.120042 6259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:44.120080 6259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:44.120139 6259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:35:44.120177 6259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:35:44.125545 6259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:39911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:44.126839 6259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:44.128086 6259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:44.137903 6267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:44.137903 6264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:44.137882 6259 server_base.cc:1034] running on GCE node
W20260928 14:35:44.137880 6265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:44.138751 6259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:44.140067 6259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:44.141234 6259 hybrid_clock.cc:648] HybridClock initialized: now 1790606144141221 us; error 30 us; skew 500 ppm
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:44.144163 6259 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:35:44.145155 6259 webserver.cc:460] Webserver started at http://127.6.23.254:34625/ using document root <none> and password file <none>
I20260928 14:35:44.145764 6259 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:44.145859 6259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:44.146101 6259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:44.148319 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "9a0abb26f4c147e98159b5ca995a90e5"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "b7bb4b91b308def4c66e127d76752eab"
server_key_iv: "fa18ad058d2872517b9c6ecff035b474"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.148922 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "9a0abb26f4c147e98159b5ca995a90e5"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "b7bb4b91b308def4c66e127d76752eab"
server_key_iv: "fa18ad058d2872517b9c6ecff035b474"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.152871 6259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:35:44.155421 6274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.156728 6259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:44.156875 6259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "9a0abb26f4c147e98159b5ca995a90e5"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "b7bb4b91b308def4c66e127d76752eab"
server_key_iv: "fa18ad058d2872517b9c6ecff035b474"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.157003 6259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:44.169231 6259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:44.172727 6259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:44.172937 6259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:44.180308 6259 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:39911
I20260928 14:35:44.180328 6326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:39911 every 8 connection(s)
I20260928 14:35:44.181282 6259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:35:44.184238 6327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:44.190047 6327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5: Bootstrap starting.
I20260928 14:35:44.190117 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 6259
I20260928 14:35:44.190269 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:35:44.191962 6239 external_mini_cluster.cc:1428] Setting key 9d9161bb9922f4deec4438575c5f0481
I20260928 14:35:44.193281 6327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:44.194229 6327 log.cc:824] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:44.196475 6327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5: No bootstrap required, opened a new log
I20260928 14:35:44.200678 6327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0abb26f4c147e98159b5ca995a90e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39911 } }
I20260928 14:35:44.200960 6327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:44.201032 6327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0abb26f4c147e98159b5ca995a90e5, State: Initialized, Role: FOLLOWER
I20260928 14:35:44.201519 6327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0abb26f4c147e98159b5ca995a90e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39911 } }
I20260928 14:35:44.201647 6327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:35:44.201737 6327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:35:44.201861 6327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:44.202800 6327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0abb26f4c147e98159b5ca995a90e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39911 } }
I20260928 14:35:44.203161 6327 leader_election.cc:302] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a0abb26f4c147e98159b5ca995a90e5; no voters:
I20260928 14:35:44.203547 6327 leader_election.cc:288] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:35:44.203675 6332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:44.203905 6332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [term 1 LEADER]: Becoming Leader. State: Replica: 9a0abb26f4c147e98159b5ca995a90e5, State: Running, Role: LEADER
I20260928 14:35:44.204226 6332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0abb26f4c147e98159b5ca995a90e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39911 } }
I20260928 14:35:44.204841 6327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:35:44.205852 6333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a0abb26f4c147e98159b5ca995a90e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0abb26f4c147e98159b5ca995a90e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39911 } } }
I20260928 14:35:44.206001 6333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:44.206301 6334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a0abb26f4c147e98159b5ca995a90e5. Latest consensus state: current_term: 1 leader_uuid: "9a0abb26f4c147e98159b5ca995a90e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a0abb26f4c147e98159b5ca995a90e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39911 } } }
I20260928 14:35:44.206369 6334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:44.206620 6341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:35:44.209936 6341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:44.216943 6341 catalog_manager.cc:1347] Generated new cluster ID: fe79589b46b84eaabe5d360211200209
I20260928 14:35:44.217033 6341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:35:44.218775 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.195596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.6.23.254:39911)
0928 14:35:44.196080 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:44.196100 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:44.197388 (+ 1288us) server_negotiation.cc:407] Connection header received
0928 14:35:44.200179 (+ 2791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:44.200200 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:44.200600 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:44.200976 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:44.203460 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.204957 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.206830 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.207611 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.207774 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.212754 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:44.212767 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:44.212779 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:44.212808 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:44.214852 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.215603 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.215609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.215615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.215776 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.216200 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.216203 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.216205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.216703 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:44.216862 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:44.217614 (+ 752us) server_negotiation.cc:299] Negotiation successful
0928 14:35:44.217903 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":148,"threads_started":1}
I20260928 14:35:44.231869 6341 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:35:44.232880 6341 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:35:44.239594 6341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a0abb26f4c147e98159b5ca995a90e5: Generated new TSK 0
I20260928 14:35:44.240458 6341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:35:44.272087 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39911
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:44.385295 6353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:44.385627 6353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:44.385706 6353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:44.389925 6353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:44.390026 6353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:44.390125 6353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:35:44.395569 6353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:44.396720 6353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:44.397782 6353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:44.407855 6361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:44.407815 6358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:44.407859 6359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:44.408483 6353 server_base.cc:1034] running on GCE node
I20260928 14:35:44.408921 6353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:44.409612 6353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:44.410800 6353 hybrid_clock.cc:648] HybridClock initialized: now 1790606144410769 us; error 49 us; skew 500 ppm
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:44.413626 6353 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:35:44.414536 6353 webserver.cc:460] Webserver started at http://127.6.23.193:36051/ using document root <none> and password file <none>
I20260928 14:35:44.415126 6353 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:44.415220 6353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:44.415482 6353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:44.417552 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "8fd53dfe20ac44ac9682bb66c8888aec"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "8ad87f63b46a864887b1c714e981e35f"
server_key_iv: "a3adb134501d5fa2cc1b772f747859d1"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.418114 6353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "8fd53dfe20ac44ac9682bb66c8888aec"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "8ad87f63b46a864887b1c714e981e35f"
server_key_iv: "a3adb134501d5fa2cc1b772f747859d1"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.421911 6353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 14:35:44.424444 6368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.425791 6353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:35:44.425887 6353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "8fd53dfe20ac44ac9682bb66c8888aec"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "8ad87f63b46a864887b1c714e981e35f"
server_key_iv: "a3adb134501d5fa2cc1b772f747859d1"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.425973 6353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:44.446301 6353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:44.449270 6353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:44.449431 6353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:44.450107 6353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:44.450958 6353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:44.451004 6353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.451056 6353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:44.451076 6353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.460429 6353 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:40361
I20260928 14:35:44.460462 6481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:40361 every 8 connection(s)
I20260928 14:35:44.461450 6353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:35:44.469142 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 6353
I20260928 14:35:44.469269 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:35:44.469524 6239 external_mini_cluster.cc:1428] Setting key a0f255499e40ac62ad9bed3ec3abc975
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:44.475859 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.463399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:42209 (local address 127.6.23.254:39911)
0928 14:35:44.463541 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:44.463545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:44.464415 (+ 870us) server_negotiation.cc:407] Connection header received
0928 14:35:44.465795 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:44.465799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:44.465855 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:44.465936 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:44.467117 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.467770 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.468945 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.469197 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.469314 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.472176 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:44.472180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:44.472183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:44.472211 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:44.473722 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.474255 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.474259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.474261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.474345 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.474689 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.474694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.474696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.474881 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:44.474988 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:44.475576 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 14:35:44.475701 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:35:44.476715 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.463816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39911 (local address 127.6.23.193:42209)
0928 14:35:44.464268 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:44.464312 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:44.465581 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:44.466110 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:44.466117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:44.466478 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:44.466959 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:44.466972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.467910 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.467919 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:44.468810 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:44.468820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.469358 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.469362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:44.469541 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.470383 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:44.470399 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:44.471999 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:44.473848 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:44.473854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:44.473870 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:44.474147 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:44.474439 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:44.474443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:44.474445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:44.474591 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:44.474988 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:44.474997 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:44.475314 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 14:35:44.475518 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 14:35:44.475858 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":122,"threads_started":1}
I20260928 14:35:44.478027 6482 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39911
I20260928 14:35:44.478296 6482 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:35:44.478905 6482 heartbeater.cc:511] Master 127.6.23.254:39911 requested a full tablet report, sending...
I20260928 14:35:44.480520 6287 ts_manager.cc:194] Registered new tserver with Master: 8fd53dfe20ac44ac9682bb66c8888aec (127.6.23.193:40361)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:35:44.481801 6287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:42209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:35:44.499105 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39911
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:44.612900 6487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:44.613188 6487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:44.613235 6487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:44.617262 6487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:44.617323 6487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:44.617384 6487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:35:44.622656 6487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:44.623773 6487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:44.624782 6487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:44.635212 6495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:44.635255 6493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:44.635219 6487 server_base.cc:1034] running on GCE node
W20260928 14:35:44.635247 6492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:44.635962 6487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:44.636650 6487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:44.637837 6487 hybrid_clock.cc:648] HybridClock initialized: now 1790606144637823 us; error 28 us; skew 500 ppm
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:44.640800 6487 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:35:44.641726 6487 webserver.cc:460] Webserver started at http://127.6.23.194:46735/ using document root <none> and password file <none>
I20260928 14:35:44.642323 6487 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:44.642414 6487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:44.642648 6487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:44.644837 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "d7b2a9971e6f4390a6cdc8bb5c1b4203"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "19a134991d73466adf52c6dee59adffd"
server_key_iv: "05517d61ee0b7d06ae60ee34c7385504"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.645448 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "d7b2a9971e6f4390a6cdc8bb5c1b4203"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "19a134991d73466adf52c6dee59adffd"
server_key_iv: "05517d61ee0b7d06ae60ee34c7385504"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.649442 6487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:35:44.652132 6502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.653398 6487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:44.653538 6487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "d7b2a9971e6f4390a6cdc8bb5c1b4203"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "19a134991d73466adf52c6dee59adffd"
server_key_iv: "05517d61ee0b7d06ae60ee34c7385504"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.653666 6487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:44.676208 6487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:44.679720 6487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:44.679925 6487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:44.680548 6487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:44.681527 6487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:44.681588 6487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.681658 6487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:44.681691 6487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.691776 6487 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:45823
I20260928 14:35:44.691803 6615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:45823 every 8 connection(s)
I20260928 14:35:44.692720 6487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:35:44.695832 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 6487
I20260928 14:35:44.695955 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:35:44.696297 6239 external_mini_cluster.cc:1428] Setting key 338b1eb337596c40f578ecf4cfb0f5d7
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:35:44.708449 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.694634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:44261 (local address 127.6.23.254:39911)
0928 14:35:44.694768 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:44.694772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:44.695740 (+ 968us) server_negotiation.cc:407] Connection header received
0928 14:35:44.697456 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:44.697459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:44.697504 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:44.697575 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:44.699143 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.699938 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.701109 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.701372 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.701492 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.704372 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:44.704376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:44.704378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:44.704403 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:44.706080 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.706708 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.706712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.706714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.706804 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.707176 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.707179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.707181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.707439 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:44.707567 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:44.708077 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 14:35:44.708232 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:35:44.709414 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.695029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39911 (local address 127.6.23.194:44261)
0928 14:35:44.695554 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:44.695633 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:44.697235 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:44.697751 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:44.697758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:44.698250 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:44.698979 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:44.699002 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.700069 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.700071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:44.700976 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:44.700984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.701533 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.701537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:44.701706 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.702381 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:44.702407 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:44.704180 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:44.706272 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:44.706278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:44.706294 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:44.706594 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:44.706901 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:44.706904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:44.706905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:44.707043 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:44.707568 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:44.707576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:44.707831 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 14:35:44.708042 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 14:35:44.708393 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":188,"threads_started":1}
I20260928 14:35:44.710776 6616 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39911
I20260928 14:35:44.711040 6616 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:44.711601 6616 heartbeater.cc:511] Master 127.6.23.254:39911 requested a full tablet report, sending...
I20260928 14:35:44.712724 6287 ts_manager.cc:194] Registered new tserver with Master: d7b2a9971e6f4390a6cdc8bb5c1b4203 (127.6.23.194:45823)
I20260928 14:35:44.713379 6287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:44261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:35:44.726776 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39911
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:35269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:44.837903 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:44.838207 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:44.838280 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:44.842391 6621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:44.842491 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:44.842588 6621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:35:44.847990 6621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:35269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39911
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:44.849064 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:44.850116 6621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:44.860080 6626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:44.860126 6627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:44.859956 6629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:44.860286 6621 server_base.cc:1034] running on GCE node
I20260928 14:35:44.860839 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:44.861557 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:44.862720 6621 hybrid_clock.cc:648] HybridClock initialized: now 1790606144862703 us; error 27 us; skew 500 ppm
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:44.865605 6621 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:35:44.866468 6621 webserver.cc:460] Webserver started at http://127.6.23.195:42335/ using document root <none> and password file <none>
I20260928 14:35:44.867041 6621 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:44.867102 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:44.867399 6621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:44.869490 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "fcd848baf1f44cc392fa6e0d8f6e765a"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "03fdcf34939f20b1fa2229aad7c24e45"
server_key_iv: "5015b9d351125e28820b0eccd8afd2e3"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.870126 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "fcd848baf1f44cc392fa6e0d8f6e765a"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "03fdcf34939f20b1fa2229aad7c24e45"
server_key_iv: "5015b9d351125e28820b0eccd8afd2e3"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.874156 6621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:35:44.876729 6636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.877982 6621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:35:44.878124 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "fcd848baf1f44cc392fa6e0d8f6e765a"
format_stamp: "Formatted at 2026-09-28 14:35:44 on dist-test-slave-06lh"
server_key: "03fdcf34939f20b1fa2229aad7c24e45"
server_key_iv: "5015b9d351125e28820b0eccd8afd2e3"
server_key_version: "encryptionkey@0"
I20260928 14:35:44.878245 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:44.894866 6621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:44.897867 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:44.898077 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:44.898677 6621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:44.899578 6621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:44.899648 6621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.899713 6621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:44.899763 6621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:44.908993 6621 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:44301
I20260928 14:35:44.909015 6749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:44301 every 8 connection(s)
I20260928 14:35:44.909886 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:35:44.913334 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 6621
I20260928 14:35:44.913479 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:35:44.913810 6239 external_mini_cluster.cc:1428] Setting key 29d7e51eb9b50a9bd0080380fde8646f
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:44.923781 6330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.911773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:36727 (local address 127.6.23.254:39911)
0928 14:35:44.911914 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:44.911918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:44.912743 (+ 825us) server_negotiation.cc:407] Connection header received
0928 14:35:44.914504 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:44.914515 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:44.914576 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:44.914682 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:44.916026 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.916678 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.917826 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.918050 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.918165 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.920628 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:44.920632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:44.920634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:44.920658 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:44.921900 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.922369 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.922371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.922373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.922444 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.922735 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.922737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.922738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.922902 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:44.922975 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:44.923546 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 14:35:44.923654 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:35:44.924803 6752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.912158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39911 (local address 127.6.23.195:36727)
0928 14:35:44.912598 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:44.912640 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:44.914267 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:44.914866 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:44.914873 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:44.915389 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:44.915882 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:44.915895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.916808 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.916812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:44.917700 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:44.917708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.918202 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.918206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:44.918380 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.918936 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:44.918952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:44.920472 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:44.922004 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:44.922009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:44.922032 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:44.922282 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:44.922521 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:44.922523 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:44.922524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:44.922657 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:44.922986 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:44.922991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:44.923292 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 14:35:44.923465 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 14:35:44.923741 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260928 14:35:44.926074 6750 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39911
I20260928 14:35:44.926335 6750 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:44.926860 6750 heartbeater.cc:511] Master 127.6.23.254:39911 requested a full tablet report, sending...
I20260928 14:35:44.928133 6291 ts_manager.cc:194] Registered new tserver with Master: fcd848baf1f44cc392fa6e0d8f6e765a (127.6.23.195:44301)
I20260928 14:35:44.928730 6291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:36727
I20260928 14:35:44.940371 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:35:44.967556 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.957811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.6.23.193:40361)
0928 14:35:44.958144 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:44.958161 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:44.958188 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:35:44.958301 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:44.958306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:44.958507 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:44.958688 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:44.959456 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.960235 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.961150 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.961545 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.961663 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.964166 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:44.964174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:44.964187 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:44.964211 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:44.965672 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.966101 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.966114 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.966116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.966196 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.966471 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.966475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.966477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.966765 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:44.967001 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:44.967174 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 14:35:44.967365 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
W20260928 14:35:44.968555 6396 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53120
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:35:44 dist-test-slave-06lh krb5kdc[6246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:35:44.993916 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:44.985822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.6.23.193:40361)
0928 14:35:44.985992 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:44.985996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:44.986018 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:35:44.986151 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:44.986153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:44.986197 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:44.986270 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:44.986932 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.987525 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.988326 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:44.988549 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:44.988623 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:44.990912 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:44.990914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:44.990917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:44.990941 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:44.992448 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.992833 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.992835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.992837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.992910 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:44.993126 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:44.993128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:44.993129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:44.993291 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:44.993389 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:44.993587 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 14:35:44.993693 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:35:44.995035 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 6353
I20260928 14:35:45.001403 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 6487
I20260928 14:35:45.009258 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 6621
I20260928 14:35:45.014954 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 6259
2026-09-28T14:35:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2200 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:35:45 dist-test-slave-06lh krb5kdc[6764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:35:45 dist-test-slave-06lh krb5kdc[6764](info): set up 2 sockets
Sep 28 14:35:45 dist-test-slave-06lh krb5kdc[6764](info): commencing operation
krb5kdc: starting...
W20260928 14:35:46.056484 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:35:46 dist-test-slave-06lh krb5kdc[6764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:35:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:35:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:35:46.179010 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40111
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:40111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:46.292145 6777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:46.292471 6777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:46.292548 6777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:46.296648 6777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:35:46.296738 6777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:46.296777 6777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:46.296838 6777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:35:46.296878 6777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:35:46.302173 6777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:40111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:46.303354 6777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:46.304388 6777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:46.312893 6782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:46.312963 6785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:46.312930 6783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:46.313102 6777 server_base.cc:1034] running on GCE node
I20260928 14:35:46.313740 6777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:46.314826 6777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:46.316020 6777 hybrid_clock.cc:648] HybridClock initialized: now 1790606146315993 us; error 38 us; skew 500 ppm
Sep 28 14:35:46 dist-test-slave-06lh krb5kdc[6764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606146, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:46.318835 6777 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:35:46.319824 6777 webserver.cc:460] Webserver started at http://127.6.23.254:37939/ using document root <none> and password file <none>
I20260928 14:35:46.320358 6777 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:46.320418 6777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:46.320595 6777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:46.322678 6777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "402164d3b9a54efda67fede294c83754"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "b88511719fc0033250e74a926b1f9c51"
server_key_iv: "f5a50264da932bee3ddd9d26170d1eb4"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.323206 6777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "402164d3b9a54efda67fede294c83754"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "b88511719fc0033250e74a926b1f9c51"
server_key_iv: "f5a50264da932bee3ddd9d26170d1eb4"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.327018 6777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:35:46.329450 6792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:46.330689 6777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:46.330789 6777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "402164d3b9a54efda67fede294c83754"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "b88511719fc0033250e74a926b1f9c51"
server_key_iv: "f5a50264da932bee3ddd9d26170d1eb4"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.330878 6777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:46.345885 6777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:46.348739 6777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:46.348898 6777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:46.356122 6777 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:40111
I20260928 14:35:46.356153 6844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:40111 every 8 connection(s)
I20260928 14:35:46.357091 6777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:35:46.359910 6845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:46.365541 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 6777
I20260928 14:35:46.365669 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:35:46.365650 6845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754: Bootstrap starting.
I20260928 14:35:46.365949 6239 external_mini_cluster.cc:1428] Setting key 92af3b5bb5ea29187acd60b84135b67b
I20260928 14:35:46.368377 6845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:46.369120 6845 log.cc:824] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:46.370973 6845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754: No bootstrap required, opened a new log
I20260928 14:35:46.374714 6845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402164d3b9a54efda67fede294c83754" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40111 } }
Sep 28 14:35:46 dist-test-slave-06lh krb5kdc[6764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:46.374956 6845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:46.375034 6845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402164d3b9a54efda67fede294c83754, State: Initialized, Role: FOLLOWER
I20260928 14:35:46.375519 6845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402164d3b9a54efda67fede294c83754" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40111 } }
I20260928 14:35:46.375694 6845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:35:46.375769 6845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:35:46.375874 6845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:46.377182 6845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402164d3b9a54efda67fede294c83754" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40111 } }
I20260928 14:35:46.377573 6845 leader_election.cc:302] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 402164d3b9a54efda67fede294c83754; no voters:
I20260928 14:35:46.377942 6845 leader_election.cc:288] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:35:46.378057 6850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:46.378286 6850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [term 1 LEADER]: Becoming Leader. State: Replica: 402164d3b9a54efda67fede294c83754, State: Running, Role: LEADER
I20260928 14:35:46.378700 6850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402164d3b9a54efda67fede294c83754" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40111 } }
I20260928 14:35:46.379308 6845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:35:46.380501 6851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "402164d3b9a54efda67fede294c83754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402164d3b9a54efda67fede294c83754" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40111 } } }
I20260928 14:35:46.380659 6851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:46.380985 6852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 402164d3b9a54efda67fede294c83754. Latest consensus state: current_term: 1 leader_uuid: "402164d3b9a54efda67fede294c83754" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402164d3b9a54efda67fede294c83754" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40111 } } }
I20260928 14:35:46.381132 6852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754 [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:46.381455 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:46.367475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.6.23.254:40111)
0928 14:35:46.367929 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:46.367949 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:46.367984 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:35:46.368708 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:46.368727 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:46.369077 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:46.369450 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:46.370244 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.371349 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.372212 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.372718 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.372893 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:46.375643 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:46.375650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:46.375667 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:46.375703 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:46.377732 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:46.378319 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:46.378327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:46.378335 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:46.378480 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:46.378810 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:46.378814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:46.378817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:46.379372 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:46.379559 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:46.379991 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 14:35:46.380344 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260928 14:35:46.381666 6859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:35:46.385020 6859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:35:46.390698 6859 catalog_manager.cc:1347] Generated new cluster ID: 73a5626e0d5345468aa44d2ae3997549
I20260928 14:35:46.390765 6859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:35:46.413679 6859 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:35:46.414707 6859 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:35:46.421499 6859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 402164d3b9a54efda67fede294c83754: Generated new TSK 0
I20260928 14:35:46.422267 6859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:35:46.453807 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40111
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:46.599893 6871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:46.600157 6871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:46.600200 6871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:46.604204 6871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:46.604260 6871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:46.604317 6871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:35:46.609519 6871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:46.610551 6871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:46.611588 6871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:46.621178 6879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:46.621300 6871 server_base.cc:1034] running on GCE node
W20260928 14:35:46.621188 6877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:46.621193 6876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:46.621901 6871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:46.622573 6871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:46.623765 6871 hybrid_clock.cc:648] HybridClock initialized: now 1790606146623739 us; error 42 us; skew 500 ppm
Sep 28 14:35:46 dist-test-slave-06lh krb5kdc[6764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606146, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:46.626441 6871 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:35:46.627385 6871 webserver.cc:460] Webserver started at http://127.6.23.193:45947/ using document root <none> and password file <none>
I20260928 14:35:46.627954 6871 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:46.628046 6871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:46.628291 6871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:46.630379 6871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "2b79f0faa817455ea6d014922ad3ee8d"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "ab546870a5e8f6ff0a14488450c5bede"
server_key_iv: "29bd5b22a7b152d11005fe2e7b1aa540"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.630954 6871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "2b79f0faa817455ea6d014922ad3ee8d"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "ab546870a5e8f6ff0a14488450c5bede"
server_key_iv: "29bd5b22a7b152d11005fe2e7b1aa540"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.634891 6871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 14:35:46.637454 6886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:46.638684 6871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:46.638820 6871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "2b79f0faa817455ea6d014922ad3ee8d"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "ab546870a5e8f6ff0a14488450c5bede"
server_key_iv: "29bd5b22a7b152d11005fe2e7b1aa540"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.638935 6871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:46.664175 6871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:46.667182 6871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:46.667441 6871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:46.668066 6871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:46.668917 6871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:46.668988 6871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:46.669071 6871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:46.669123 6871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:46.677956 6871 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:43933
I20260928 14:35:46.677982 6999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:43933 every 8 connection(s)
I20260928 14:35:46.678871 6871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:35:46.681360 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 6871
I20260928 14:35:46.681511 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:35:46.681839 6239 external_mini_cluster.cc:1428] Setting key 817e425a8fc2dcd5203e62ae7aef94f4
Sep 28 14:35:46 dist-test-slave-06lh krb5kdc[6764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606146, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:35:46.696730 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:46.681221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:58113 (local address 127.6.23.254:40111)
0928 14:35:46.681383 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:46.681390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:46.682720 (+ 1330us) server_negotiation.cc:407] Connection header received
0928 14:35:46.684649 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:46.684679 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:46.684745 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:46.684831 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:46.686615 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.687667 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.689465 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.689922 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.690057 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:46.692904 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:46.692912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:46.692914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:46.692939 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:46.694423 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:46.695041 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:46.695048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:46.695065 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:46.695169 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:46.695629 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:46.695633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:46.695635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:46.695824 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:46.695934 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:46.696438 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 14:35:46.696565 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:35:46.697618 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:46.681764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40111 (local address 127.6.23.193:58113)
0928 14:35:46.682488 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:46.682609 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:46.684378 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:46.685162 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:46.685174 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:46.685726 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:46.686424 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:46.686451 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.687802 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.687806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:46.689320 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:46.689334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.690125 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.690129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:46.690329 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:46.690963 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:46.690979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:46.692722 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:46.694531 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:46.694537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:46.694551 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:46.694941 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:46.695355 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:46.695358 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:46.695360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:46.695515 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:46.695932 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:46.695939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:46.696200 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 14:35:46.696395 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 14:35:46.696750 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":170,"threads_started":1}
I20260928 14:35:46.699023 7000 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40111
I20260928 14:35:46.699335 7000 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:46.699841 7000 heartbeater.cc:511] Master 127.6.23.254:40111 requested a full tablet report, sending...
I20260928 14:35:46.701426 6809 ts_manager.cc:194] Registered new tserver with Master: 2b79f0faa817455ea6d014922ad3ee8d (127.6.23.193:43933)
I20260928 14:35:46.702570 6809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:58113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:35:46.710354 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40111
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:46.821022 7005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:46.821316 7005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:46.821396 7005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:46.825460 7005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:46.825562 7005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:46.825659 7005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:35:46.831069 7005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:46.832237 7005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:46.833269 7005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:46.842794 7010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:46.842803 7013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:46.842803 7011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:46.843092 7005 server_base.cc:1034] running on GCE node
I20260928 14:35:46.843550 7005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:46.844204 7005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:46.845383 7005 hybrid_clock.cc:648] HybridClock initialized: now 1790606146845348 us; error 47 us; skew 500 ppm
Sep 28 14:35:46 dist-test-slave-06lh krb5kdc[6764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606146, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:46.847995 7005 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:35:46.848873 7005 webserver.cc:460] Webserver started at http://127.6.23.194:37559/ using document root <none> and password file <none>
I20260928 14:35:46.849427 7005 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:46.849525 7005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:46.849761 7005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:46.851830 7005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "3a57fa90528f40b4a67211c08e2cd313"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "784b861e66b292bd041c12c2533cb030"
server_key_iv: "2b4c6f0ea9f7ec4ebd068ac2dacbbe41"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.852397 7005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "3a57fa90528f40b4a67211c08e2cd313"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "784b861e66b292bd041c12c2533cb030"
server_key_iv: "2b4c6f0ea9f7ec4ebd068ac2dacbbe41"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.856056 7005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:35:46.858374 7020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:46.859629 7005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:46.859769 7005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "3a57fa90528f40b4a67211c08e2cd313"
format_stamp: "Formatted at 2026-09-28 14:35:46 on dist-test-slave-06lh"
server_key: "784b861e66b292bd041c12c2533cb030"
server_key_iv: "2b4c6f0ea9f7ec4ebd068ac2dacbbe41"
server_key_version: "encryptionkey@0"
I20260928 14:35:46.859880 7005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:46.869822 7005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:46.872895 7005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:46.873111 7005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:46.873720 7005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:46.874560 7005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:46.874630 7005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:46.874701 7005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:46.874744 7005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:46.883998 7005 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:37169
I20260928 14:35:46.884057 7133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:37169 every 8 connection(s)
I20260928 14:35:46.884922 7005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:35:46.886644 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 7005
I20260928 14:35:46.886754 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:35:46.887051 6239 external_mini_cluster.cc:1428] Setting key 5261ac344c98b8972e3638e879169a1a
Sep 28 14:35:46 dist-test-slave-06lh krb5kdc[6764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606146, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:35:46.900787 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:46.887375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:48579 (local address 127.6.23.254:40111)
0928 14:35:46.887586 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:46.887591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:46.888301 (+ 710us) server_negotiation.cc:407] Connection header received
0928 14:35:46.889803 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:46.889809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:46.889890 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:46.889991 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:46.891637 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.892533 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.893731 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.894177 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.894315 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:46.897102 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:46.897109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:46.897114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:46.897144 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:46.898510 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:46.899059 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:46.899061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:46.899063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:46.899170 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:46.899604 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:46.899610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:46.899615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:46.899819 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:46.899903 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:46.900562 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 14:35:46.900664 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 14:35:46.901855 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:46.887637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40111 (local address 127.6.23.194:48579)
0928 14:35:46.888137 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:46.888189 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:46.889592 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:46.890212 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:46.890223 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:46.890742 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:46.891485 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:46.891502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.892678 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.892681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:46.893596 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:46.893605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:46.894332 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:46.894334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:46.894477 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:46.895091 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:46.895108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:46.896920 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:46.898673 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:46.898678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:46.898690 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:46.898949 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:46.899308 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:46.899310 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:46.899312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:46.899495 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:46.899936 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:46.899946 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:46.900288 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 14:35:46.900501 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 14:35:46.900990 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":134,"threads_started":1}
I20260928 14:35:46.903132 7134 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40111
I20260928 14:35:46.903450 7134 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:46.904114 7134 heartbeater.cc:511] Master 127.6.23.254:40111 requested a full tablet report, sending...
I20260928 14:35:46.905409 6809 ts_manager.cc:194] Registered new tserver with Master: 3a57fa90528f40b4a67211c08e2cd313 (127.6.23.194:37169)
I20260928 14:35:46.905997 6809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:48579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:35:46.917928 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40111
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:40887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:47.028445 7139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:47.028780 7139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:47.028861 7139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:47.033037 7139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:47.033126 7139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:47.033218 7139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:35:47.038839 7139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40111
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:47.039976 7139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:47.041018 7139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:47.050613 7144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:47.050644 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:47.050670 7145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:47.050840 7139 server_base.cc:1034] running on GCE node
I20260928 14:35:47.051412 7139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:47.052064 7139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:47.053249 7139 hybrid_clock.cc:648] HybridClock initialized: now 1790606147053233 us; error 30 us; skew 500 ppm
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[6764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606147, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:47.056054 7139 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:35:47.056946 7139 webserver.cc:460] Webserver started at http://127.6.23.195:42697/ using document root <none> and password file <none>
I20260928 14:35:47.057507 7139 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:47.057600 7139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:47.057825 7139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:47.059948 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "56f67e2096394094bbe6f900897505ba"
format_stamp: "Formatted at 2026-09-28 14:35:47 on dist-test-slave-06lh"
server_key: "ecbe2182c57d88250208311b970ba296"
server_key_iv: "5cea442737599637f90c26421879d40d"
server_key_version: "encryptionkey@0"
I20260928 14:35:47.060498 7139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "56f67e2096394094bbe6f900897505ba"
format_stamp: "Formatted at 2026-09-28 14:35:47 on dist-test-slave-06lh"
server_key: "ecbe2182c57d88250208311b970ba296"
server_key_iv: "5cea442737599637f90c26421879d40d"
server_key_version: "encryptionkey@0"
I20260928 14:35:47.064312 7139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:35:47.066743 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:47.068002 7139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:47.068140 7139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "56f67e2096394094bbe6f900897505ba"
format_stamp: "Formatted at 2026-09-28 14:35:47 on dist-test-slave-06lh"
server_key: "ecbe2182c57d88250208311b970ba296"
server_key_iv: "5cea442737599637f90c26421879d40d"
server_key_version: "encryptionkey@0"
I20260928 14:35:47.068238 7139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:47.096221 7139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:47.099324 7139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:47.099524 7139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:47.100138 7139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:47.100955 7139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:47.101001 7139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:47.101095 7139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:47.101150 7139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:47.109773 7139 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:37163
I20260928 14:35:47.109843 7267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:37163 every 8 connection(s)
I20260928 14:35:47.110720 7139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:35:47.114183 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 7139
I20260928 14:35:47.114343 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:35:47.114650 6239 external_mini_cluster.cc:1428] Setting key c6940ba8ef57a20f28221b31bd2188bc
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[6764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606147, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:47.124701 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.112557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:51185 (local address 127.6.23.254:40111)
0928 14:35:47.112708 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.112712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.113594 (+ 882us) server_negotiation.cc:407] Connection header received
0928 14:35:47.115357 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.115361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.115417 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.115508 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:47.117036 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.117557 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.118662 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.118894 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.118981 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.121345 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:47.121348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:47.121350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:47.121373 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:47.122838 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.123339 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.123342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.123346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.123428 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.123749 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.123751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.123752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.123930 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:47.124004 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.124474 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.124578 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:35:47.125507 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.113011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40111 (local address 127.6.23.195:51185)
0928 14:35:47.113443 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:47.113485 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:47.115106 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:47.115720 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:47.115729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:47.116229 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:47.116894 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.116911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.117658 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.117660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.118510 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.118519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.119024 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.119027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.119167 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.119763 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:47.119779 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:47.121195 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:47.122931 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:47.122935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:47.122946 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:47.123178 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:47.123513 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:47.123515 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:47.123516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:47.123657 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:47.124013 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:47.124017 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:47.124233 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 14:35:47.124379 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 14:35:47.124631 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":114,"threads_started":1}
I20260928 14:35:47.126668 7268 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40111
I20260928 14:35:47.126924 7268 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:47.127558 7268 heartbeater.cc:511] Master 127.6.23.254:40111 requested a full tablet report, sending...
I20260928 14:35:47.128579 6809 ts_manager.cc:194] Registered new tserver with Master: 56f67e2096394094bbe6f900897505ba (127.6.23.195:37163)
I20260928 14:35:47.129146 6809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:51185
I20260928 14:35:47.140482 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:35:47.151168 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.144159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.6.23.254:40111)
0928 14:35:47.144301 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.144305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.144465 (+ 160us) server_negotiation.cc:407] Connection header received
0928 14:35:47.144592 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.144594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.144634 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.144699 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:47.145433 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.145953 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.146723 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.146931 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.147009 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.148314 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:47.148317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:47.148319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:47.148342 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:47.149782 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.150123 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.150125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.150127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.150195 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.150407 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.150409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.150410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.150610 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:47.150708 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.150882 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.150966 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:35:47.156070 6809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:35:47.159096 6809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:47.174067 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.167596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:43933 (local address 127.0.0.1:38080)
0928 14:35:47.167935 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:47.167981 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:47.168102 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:47.169216 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:47.169222 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:47.169260 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:47.169516 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.169521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.171080 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.171083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.172408 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.172418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.173482 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.173485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.173608 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.173649 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:35:47.173763 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:35:47.173883 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":152,"spinlock_wait_cycles":7552,"thread_start_us":102,"threads_started":1}
I20260928 14:35:47.174091 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.167737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.6.23.193:43933)
0928 14:35:47.168200 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.168216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.168240 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:35:47.168321 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.168328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.168539 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.169573 (+ 1034us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:47.169738 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.170948 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.172514 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.173354 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.173470 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.173618 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.173809 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.173921 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":105,"threads_started":1}
I20260928 14:35:47.174095 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.167891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37163 (local address 127.0.0.1:42264)
0928 14:35:47.168333 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:47.168345 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:47.168430 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:47.169226 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:47.169228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:47.169258 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:47.169498 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.169506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.170651 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.170657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.172302 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.172313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.173686 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.173689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.173807 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.173831 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:35:47.173902 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:35:47.173971 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":105,"spinlock_wait_cycles":5760,"thread_start_us":82,"threads_started":1}
I20260928 14:35:47.174393 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.168400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.6.23.195:37163)
0928 14:35:47.168750 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.168767 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.168792 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:35:47.168869 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.168873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.169083 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.169646 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:47.169791 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.170521 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.172444 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.173354 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.173470 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.173618 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.174159 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.174258 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":154,"threads_started":1}
I20260928 14:35:47.175045 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.168350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37169 (local address 127.0.0.1:55438)
0928 14:35:47.168875 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:47.168886 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:47.168964 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:47.169636 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:47.169639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:47.169657 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:47.169727 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.169731 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.172098 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.172101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.173105 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.173114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.174328 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.174332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.174743 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.174761 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:35:47.174850 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:35:47.174918 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260928 14:35:47.175110 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.168848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.6.23.194:37169)
0928 14:35:47.169126 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.169144 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.169167 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:35:47.169238 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.169242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.169415 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.169572 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:47.169836 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.170930 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.173226 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.173981 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.174106 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.174697 (+ 591us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.174880 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.174965 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":118,"threads_started":1}
I20260928 14:35:47.177196 6934 tablet_service.cc:1467] Processing CreateTablet for tablet c3a78fc14e354b3aaa7db87e2823712f (DEFAULT_TABLE table=test-table [id=e2d1c5c13ad541daa83808a31a12db84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:47.178174 6934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a78fc14e354b3aaa7db87e2823712f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:47.181480 7202 tablet_service.cc:1467] Processing CreateTablet for tablet c3a78fc14e354b3aaa7db87e2823712f (DEFAULT_TABLE table=test-table [id=e2d1c5c13ad541daa83808a31a12db84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:47.181939 7068 tablet_service.cc:1467] Processing CreateTablet for tablet c3a78fc14e354b3aaa7db87e2823712f (DEFAULT_TABLE table=test-table [id=e2d1c5c13ad541daa83808a31a12db84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:47.182542 7202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a78fc14e354b3aaa7db87e2823712f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:47.182917 7068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a78fc14e354b3aaa7db87e2823712f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:47.185804 7285 tablet_bootstrap.cc:492] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d: Bootstrap starting.
I20260928 14:35:47.188549 7285 tablet_bootstrap.cc:654] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:47.189220 7286 tablet_bootstrap.cc:492] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313: Bootstrap starting.
I20260928 14:35:47.190095 7287 tablet_bootstrap.cc:492] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba: Bootstrap starting.
I20260928 14:35:47.190790 7285 log.cc:824] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:47.191847 7286 tablet_bootstrap.cc:654] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:47.192080 7287 tablet_bootstrap.cc:654] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:47.192596 7286 log.cc:824] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:47.193099 7285 tablet_bootstrap.cc:492] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d: No bootstrap required, opened a new log
I20260928 14:35:47.193243 7287 log.cc:824] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:47.193305 7285 ts_tablet_manager.cc:1397] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 14:35:47.194389 7286 tablet_bootstrap.cc:492] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313: No bootstrap required, opened a new log
I20260928 14:35:47.194594 7286 ts_tablet_manager.cc:1397] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:35:47.195125 7287 tablet_bootstrap.cc:492] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba: No bootstrap required, opened a new log
I20260928 14:35:47.195329 7287 ts_tablet_manager.cc:1397] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:35:47.197543 7285 raft_consensus.cc:348] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.197822 7285 raft_consensus.cc:374] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:47.197886 7285 raft_consensus.cc:729] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b79f0faa817455ea6d014922ad3ee8d, State: Initialized, Role: FOLLOWER
I20260928 14:35:47.197702 7286 raft_consensus.cc:348] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.197968 7286 raft_consensus.cc:374] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:47.198028 7286 raft_consensus.cc:729] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a57fa90528f40b4a67211c08e2cd313, State: Initialized, Role: FOLLOWER
I20260928 14:35:47.198423 7286 consensus_queue.cc:260] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.198444 7285 consensus_queue.cc:260] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.199086 7000 heartbeater.cc:503] Master 127.6.23.254:40111 was elected leader, sending a full tablet report...
I20260928 14:35:47.198966 7287 raft_consensus.cc:348] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.199195 7287 raft_consensus.cc:374] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:47.199266 7287 raft_consensus.cc:729] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f67e2096394094bbe6f900897505ba, State: Initialized, Role: FOLLOWER
I20260928 14:35:47.199445 7286 ts_tablet_manager.cc:1428] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:35:47.199482 7285 ts_tablet_manager.cc:1428] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:35:47.199754 7134 heartbeater.cc:503] Master 127.6.23.254:40111 was elected leader, sending a full tablet report...
I20260928 14:35:47.200646 7287 consensus_queue.cc:260] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.201299 7268 heartbeater.cc:503] Master 127.6.23.254:40111 was elected leader, sending a full tablet report...
I20260928 14:35:47.201584 7287 ts_tablet_manager.cc:1428] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:35:47.311868 7291 raft_consensus.cc:482] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:47.312117 7291 raft_consensus.cc:504] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.313360 7291 leader_election.cc:288] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a57fa90528f40b4a67211c08e2cd313 (127.6.23.194:37169), 56f67e2096394094bbe6f900897505ba (127.6.23.195:37163)
I20260928 14:35:47.317325 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.313762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:34663 (local address 127.6.23.195:37163)
0928 14:35:47.313886 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.313891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.314159 (+ 268us) server_negotiation.cc:407] Connection header received
0928 14:35:47.314314 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.314316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.314365 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.314450 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:47.314727 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.315220 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.316394 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.316789 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.316917 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.316969 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.317137 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.317200 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:35:47.317344 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.313612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37163 (local address 127.6.23.193:34663)
0928 14:35:47.314064 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:47.314075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:47.314209 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:47.314462 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:47.314465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:47.314481 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:47.314620 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.314625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.315358 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.315363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.316270 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.316280 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.316918 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.316922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.317006 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.317037 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:35:47.317123 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:35:47.317195 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":85,"thread_start_us":150,"threads_started":1}
I20260928 14:35:47.317472 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.313701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:33503 (local address 127.6.23.194:37169)
0928 14:35:47.313836 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.313841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.314127 (+ 286us) server_negotiation.cc:407] Connection header received
0928 14:35:47.314295 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.314302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.314348 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.314437 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:47.314712 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.315205 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.316365 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.316804 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.316914 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.316973 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.317284 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.317360 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:35:47.317869 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.313596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37169 (local address 127.6.23.193:33503)
0928 14:35:47.314011 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:47.314023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:47.314172 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:47.314447 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:47.314450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:47.314465 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:47.314604 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.314609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.315350 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.315357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.316240 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:47.316249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.317032 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.317036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:47.317143 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.317170 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:35:47.317646 (+ 476us) client_negotiation.cc:240] Negotiation successful
0928 14:35:47.317729 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":23,"spinlock_wait_cycles":896,"thread_start_us":160,"threads_started":1}
I20260928 14:35:47.318209 7222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a78fc14e354b3aaa7db87e2823712f" candidate_uuid: "2b79f0faa817455ea6d014922ad3ee8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f67e2096394094bbe6f900897505ba" is_pre_election: true
I20260928 14:35:47.318460 7088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a78fc14e354b3aaa7db87e2823712f" candidate_uuid: "2b79f0faa817455ea6d014922ad3ee8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a57fa90528f40b4a67211c08e2cd313" is_pre_election: true
I20260928 14:35:47.318527 7222 raft_consensus.cc:2454] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b79f0faa817455ea6d014922ad3ee8d in term 0.
I20260928 14:35:47.318750 7088 raft_consensus.cc:2454] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b79f0faa817455ea6d014922ad3ee8d in term 0.
I20260928 14:35:47.319051 6888 leader_election.cc:302] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b79f0faa817455ea6d014922ad3ee8d, 56f67e2096394094bbe6f900897505ba; no voters:
I20260928 14:35:47.319345 7291 raft_consensus.cc:2781] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:47.319434 7291 raft_consensus.cc:482] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:47.319478 7291 raft_consensus.cc:3037] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:47.320497 7291 raft_consensus.cc:504] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.320861 7291 leader_election.cc:288] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [CANDIDATE]: Term 1 election: Requested vote from peers 3a57fa90528f40b4a67211c08e2cd313 (127.6.23.194:37169), 56f67e2096394094bbe6f900897505ba (127.6.23.195:37163)
I20260928 14:35:47.321146 7088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a78fc14e354b3aaa7db87e2823712f" candidate_uuid: "2b79f0faa817455ea6d014922ad3ee8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a57fa90528f40b4a67211c08e2cd313"
I20260928 14:35:47.321291 7088 raft_consensus.cc:3037] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:47.321282 7222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3a78fc14e354b3aaa7db87e2823712f" candidate_uuid: "2b79f0faa817455ea6d014922ad3ee8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f67e2096394094bbe6f900897505ba"
I20260928 14:35:47.321420 7222 raft_consensus.cc:3037] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:47.322641 7222 raft_consensus.cc:2454] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b79f0faa817455ea6d014922ad3ee8d in term 1.
I20260928 14:35:47.322641 7088 raft_consensus.cc:2454] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b79f0faa817455ea6d014922ad3ee8d in term 1.
I20260928 14:35:47.322978 6889 leader_election.cc:302] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b79f0faa817455ea6d014922ad3ee8d, 3a57fa90528f40b4a67211c08e2cd313; no voters:
I20260928 14:35:47.323158 7291 raft_consensus.cc:2781] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:47.323395 7291 raft_consensus.cc:686] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [term 1 LEADER]: Becoming Leader. State: Replica: 2b79f0faa817455ea6d014922ad3ee8d, State: Running, Role: LEADER
I20260928 14:35:47.323729 7291 consensus_queue.cc:237] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } }
I20260928 14:35:47.326921 6808 catalog_manager.cc:5495] T c3a78fc14e354b3aaa7db87e2823712f P 2b79f0faa817455ea6d014922ad3ee8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b79f0faa817455ea6d014922ad3ee8d (127.6.23.193). New cstate: current_term: 1 leader_uuid: "2b79f0faa817455ea6d014922ad3ee8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b79f0faa817455ea6d014922ad3ee8d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 43933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a57fa90528f40b4a67211c08e2cd313" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56f67e2096394094bbe6f900897505ba" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37163 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[6764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
W20260928 14:35:47.361747 7269 tablet.cc:2369] T c3a78fc14e354b3aaa7db87e2823712f P 56f67e2096394094bbe6f900897505ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:35:47.365772 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.356196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.6.23.193:43933)
0928 14:35:47.356343 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.356348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.356542 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:35:47.356683 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.356686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.356743 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.356835 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:47.357558 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.358152 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.359006 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.359280 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.359403 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.362132 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:47.362147 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:47.362164 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:47.362192 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:47.364041 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.364557 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.364562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.364563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.364654 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.364925 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.364927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.364928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.365153 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:47.365305 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.365496 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.365601 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[6764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260928 14:35:47.386089 7135 tablet.cc:2369] T c3a78fc14e354b3aaa7db87e2823712f P 3a57fa90528f40b4a67211c08e2cd313: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[6764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606147, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:35:47.392375 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.384219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.6.23.193:43933)
0928 14:35:47.384410 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.384414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.384503 (+ 89us) server_negotiation.cc:407] Connection header received
0928 14:35:47.384619 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.384622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.384668 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.384738 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:47.385369 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.385929 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.386786 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.387014 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.387100 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.389369 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:47.389372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:47.389374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:47.389398 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:47.390816 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.391184 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.391187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.391188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.391322 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.391559 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.391561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.391563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.391723 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:47.391824 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.392064 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.392160 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[6764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[6764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:35:47.417693 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:47.409691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.6.23.193:43933)
0928 14:35:47.409863 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:47.409866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:47.409929 (+ 63us) server_negotiation.cc:407] Connection header received
0928 14:35:47.410041 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:47.410044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:47.410091 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:47.410168 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:47.410814 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.411383 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.412180 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:47.412390 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:47.412464 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:47.414790 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:47.414793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:47.414796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:47.414821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:47.416275 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.416644 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.416646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.416648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.416720 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:47.416938 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:47.416940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:47.416942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:47.417099 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:47.417210 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:47.417378 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 14:35:47.417470 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 14:35:47.419117 6914 tablet_service.cc:2908] Rejecting scan request for tablet c3a78fc14e354b3aaa7db87e2823712f: Uninitialized: clean time has not yet been initialized
I20260928 14:35:47.421267 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 6871
I20260928 14:35:47.427433 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 7005
I20260928 14:35:47.432833 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 7139
I20260928 14:35:47.438490 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 6777
2026-09-28T14:35:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2424 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[7310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[7310](info): set up 2 sockets
Sep 28 14:35:47 dist-test-slave-06lh krb5kdc[7310](info): commencing operation
krb5kdc: starting...
W20260928 14:35:48.500480 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:35:48 dist-test-slave-06lh krb5kdc[7310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:35:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:35:48.626629 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:38123
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:38123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:48.743899 7323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:48.744205 7323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:48.744277 7323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:48.748433 7323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:35:48.748526 7323 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:35:48.748559 7323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:48.748601 7323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:48.748636 7323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:35:48.748670 7323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:35:48.754036 7323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:38123
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:38123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:48.755206 7323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:48.756282 7323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:48.764988 7329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:48.765117 7328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:48.764997 7331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:48.765008 7323 server_base.cc:1034] running on GCE node
I20260928 14:35:48.765841 7323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:48.766953 7323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:48.768146 7323 hybrid_clock.cc:648] HybridClock initialized: now 1790606148768102 us; error 55 us; skew 500 ppm
Sep 28 14:35:48 dist-test-slave-06lh krb5kdc[7310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606148, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:48.776687 7323 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:35:48.777652 7323 webserver.cc:460] Webserver started at http://127.6.23.254:35139/ using document root <none> and password file <none>
I20260928 14:35:48.778200 7323 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:48.778259 7323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:48.778427 7323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:48.780527 7323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "aa6ad0d1edc34018b7674dfb7741e3f7"
format_stamp: "Formatted at 2026-09-28 14:35:48 on dist-test-slave-06lh"
server_key: "6add3dcc218ae6c8fca289ab61f1f789"
server_key_iv: "5f78a78098daa89e3985d5a6629e8671"
server_key_version: "encryptionkey@0"
I20260928 14:35:48.781054 7323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "aa6ad0d1edc34018b7674dfb7741e3f7"
format_stamp: "Formatted at 2026-09-28 14:35:48 on dist-test-slave-06lh"
server_key: "6add3dcc218ae6c8fca289ab61f1f789"
server_key_iv: "5f78a78098daa89e3985d5a6629e8671"
server_key_version: "encryptionkey@0"
I20260928 14:35:48.784864 7323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 14:35:48.787370 7338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:48.788612 7323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:48.788739 7323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "aa6ad0d1edc34018b7674dfb7741e3f7"
format_stamp: "Formatted at 2026-09-28 14:35:48 on dist-test-slave-06lh"
server_key: "6add3dcc218ae6c8fca289ab61f1f789"
server_key_iv: "5f78a78098daa89e3985d5a6629e8671"
server_key_version: "encryptionkey@0"
I20260928 14:35:48.788843 7323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:48.818747 7323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:48.821897 7323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:48.822115 7323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:48.830416 7323 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:38123
I20260928 14:35:48.830437 7400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:38123 every 8 connection(s)
I20260928 14:35:48.831465 7323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:35:48.833437 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 7323
I20260928 14:35:48.833523 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:35:48.833802 6239 external_mini_cluster.cc:1428] Setting key 40f717e60ba0cce2d688a3814bdbdda3
I20260928 14:35:48.834383 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:48.840507 7401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7: Bootstrap starting.
Sep 28 14:35:48 dist-test-slave-06lh krb5kdc[7310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:48.842967 7401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:48.844332 7401 log.cc:824] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:48.846642 7401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7: No bootstrap required, opened a new log
I20260928 14:35:48.847973 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:48.835253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.6.23.254:38123)
0928 14:35:48.835685 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:48.835702 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:48.835738 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:35:48.836480 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:48.836518 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:48.836862 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:48.837200 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:48.838002 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:48.839004 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:48.839931 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:48.840409 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:48.840581 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:48.843010 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:48.843015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:48.843029 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:48.843064 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:48.844987 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:48.845522 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:48.845527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:48.845532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:48.845643 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:48.845929 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:48.845932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:48.845933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:48.846334 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:48.846478 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:48.846859 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 14:35:48.847183 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":130,"threads_started":1}
I20260928 14:35:48.849829 7401 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6ad0d1edc34018b7674dfb7741e3f7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38123 } }
I20260928 14:35:48.850109 7401 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:48.850179 7401 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6ad0d1edc34018b7674dfb7741e3f7, State: Initialized, Role: FOLLOWER
I20260928 14:35:48.850598 7401 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6ad0d1edc34018b7674dfb7741e3f7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38123 } }
I20260928 14:35:48.850739 7401 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:35:48.850802 7401 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:35:48.850929 7401 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:48.851881 7401 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6ad0d1edc34018b7674dfb7741e3f7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38123 } }
I20260928 14:35:48.852228 7401 leader_election.cc:302] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa6ad0d1edc34018b7674dfb7741e3f7; no voters:
I20260928 14:35:48.852533 7401 leader_election.cc:288] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:35:48.852669 7406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:48.852896 7406 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [term 1 LEADER]: Becoming Leader. State: Replica: aa6ad0d1edc34018b7674dfb7741e3f7, State: Running, Role: LEADER
I20260928 14:35:48.853235 7406 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6ad0d1edc34018b7674dfb7741e3f7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38123 } }
I20260928 14:35:48.853753 7401 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:35:48.855103 7407 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa6ad0d1edc34018b7674dfb7741e3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6ad0d1edc34018b7674dfb7741e3f7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38123 } } }
I20260928 14:35:48.855095 7408 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa6ad0d1edc34018b7674dfb7741e3f7. Latest consensus state: current_term: 1 leader_uuid: "aa6ad0d1edc34018b7674dfb7741e3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6ad0d1edc34018b7674dfb7741e3f7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38123 } } }
I20260928 14:35:48.855255 7407 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:48.855262 7408 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:48.855707 7415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:35:48.858652 7415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:35:48.864126 7415 catalog_manager.cc:1347] Generated new cluster ID: 140084515cac4cac8db9f6af95a74723
I20260928 14:35:48.864197 7415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:35:48.884238 7415 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:35:48.885205 7415 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:35:48.890901 7415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7: Generated new TSK 0
I20260928 14:35:48.891664 7415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:35:48.925827 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38123
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:49.038832 7427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:49.039311 7427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:49.039400 7427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:49.043597 7427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:49.043684 7427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:49.043752 7427 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:35:49.043839 7427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:35:49.049263 7427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:38123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:49.050396 7427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:49.051534 7427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:49.061623 7435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:49.061619 7432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:49.061615 7433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:49.061918 7427 server_base.cc:1034] running on GCE node
I20260928 14:35:49.062495 7427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:49.063171 7427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:49.064379 7427 hybrid_clock.cc:648] HybridClock initialized: now 1790606149064347 us; error 50 us; skew 500 ppm
Sep 28 14:35:49 dist-test-slave-06lh krb5kdc[7310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:49.067354 7427 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:35:49.068300 7427 webserver.cc:460] Webserver started at http://127.6.23.193:37687/ using document root <none> and password file <none>
I20260928 14:35:49.068881 7427 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:49.068949 7427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:49.069172 7427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:49.071308 7427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "ccb81779fc954885bfdde1411fcb202d"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "8ccc04a8839613557d630663e1b460f8"
server_key_iv: "289b284401fbe675b82c1d8262e6f125"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.071959 7427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "ccb81779fc954885bfdde1411fcb202d"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "8ccc04a8839613557d630663e1b460f8"
server_key_iv: "289b284401fbe675b82c1d8262e6f125"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.075858 7427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:35:49.078377 7442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:49.079711 7427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:49.079854 7427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "ccb81779fc954885bfdde1411fcb202d"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "8ccc04a8839613557d630663e1b460f8"
server_key_iv: "289b284401fbe675b82c1d8262e6f125"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.079974 7427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:49.092837 7427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:49.095899 7427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:49.096118 7427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:49.097566 7427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:49.097641 7427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:49.097725 7427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:49.097770 7427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:35:49 dist-test-slave-06lh krb5kdc[7310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:49.109087 7427 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:36147
I20260928 14:35:49.109117 7557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:36147 every 8 connection(s)
I20260928 14:35:49.110267 7427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:35:49.111667 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.098443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:53939 (local address 127.6.23.254:38123)
0928 14:35:49.098636 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.098642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.099504 (+ 862us) server_negotiation.cc:407] Connection header received
0928 14:35:49.100470 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.100473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.100534 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.100621 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:49.102356 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.103103 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.104301 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.104552 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.104672 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.107313 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:49.107317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:49.107320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:49.107355 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:49.109114 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.109733 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.109736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.109737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.109820 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.110173 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.110176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.110177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.110407 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:49.110485 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.111387 (+ 902us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.111513 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:35:49.112324 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.098848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38123 (local address 127.6.23.193:53939)
0928 14:35:49.099344 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.099395 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.100244 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.100834 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.100844 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.101387 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:49.102149 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.102174 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.103256 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.103259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.104161 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.104177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.104736 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.104741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.104939 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.105576 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:49.105602 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:49.107099 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:49.109251 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.109257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.109272 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.109614 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.109915 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.109919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.109921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.110082 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.110492 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:49.110498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:49.110859 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.111079 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.111444 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":136,"threads_started":1}
I20260928 14:35:49.112690 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 7427
I20260928 14:35:49.112820 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:35:49.113111 6239 external_mini_cluster.cc:1428] Setting key a6e62e82a9bc397f57492c49cb9e4ad2
I20260928 14:35:49.113758 7558 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38123
I20260928 14:35:49.113907 7558 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:49.114478 7558 heartbeater.cc:511] Master 127.6.23.254:38123 requested a full tablet report, sending...
I20260928 14:35:49.116233 7354 ts_manager.cc:194] Registered new tserver with Master: ccb81779fc954885bfdde1411fcb202d (127.6.23.193:36147)
I20260928 14:35:49.117476 7354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:53939
I20260928 14:35:49.124531 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.115997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38123 (local address 127.6.23.193:54789)
0928 14:35:49.116301 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.116314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.116465 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.116829 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.116833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.117084 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:49.117232 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.117242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.118409 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.118415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.119514 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.119528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.120097 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.120104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.120205 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.120730 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:49.120741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:49.121072 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:49.123021 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.123024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.123026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.123216 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.123559 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.123562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.123563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.123639 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.124097 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:49.124099 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:49.124173 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.124252 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.124360 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:35:49.124682 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.116043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:54789 (local address 127.6.23.254:38123)
0928 14:35:49.116161 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.116165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.116398 (+ 233us) server_negotiation.cc:407] Connection header received
0928 14:35:49.116576 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.116579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.116624 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.116696 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:49.117377 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.118225 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.119695 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.119961 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.120050 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.121232 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:49.121238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:49.121243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:49.121277 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:49.122906 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.123361 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.123367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.123372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.123458 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.123741 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.123744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.123746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.123994 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:49.124114 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.124339 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.124464 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:35:49.142994 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38123
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:49.256345 7567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:49.256662 7567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:49.256749 7567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:49.260954 7567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:49.261037 7567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:49.261109 7567 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:35:49.261194 7567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:35:49.266534 7567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:38123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:49.267771 7567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:49.268834 7567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:35:49.278383 7567 server_base.cc:1034] running on GCE node
W20260928 14:35:49.278316 7573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:49.278308 7575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:49.278288 7572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:49.279021 7567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:49.279767 7567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:49.280927 7567 hybrid_clock.cc:648] HybridClock initialized: now 1790606149280908 us; error 32 us; skew 500 ppm
Sep 28 14:35:49 dist-test-slave-06lh krb5kdc[7310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:49.283625 7567 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:35:49.284543 7567 webserver.cc:460] Webserver started at http://127.6.23.194:37857/ using document root <none> and password file <none>
I20260928 14:35:49.285120 7567 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:49.285213 7567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:49.285422 7567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:49.287540 7567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "235a96d3782c49e1a85e49a98895dbbd"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "6a3fd665b34a5979748cff01d8f620ce"
server_key_iv: "a2d22f42b6ae30429659271b2edc8548"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.288120 7567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "235a96d3782c49e1a85e49a98895dbbd"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "6a3fd665b34a5979748cff01d8f620ce"
server_key_iv: "a2d22f42b6ae30429659271b2edc8548"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.291749 7567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:35:49.294178 7582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:49.295424 7567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:49.295554 7567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "235a96d3782c49e1a85e49a98895dbbd"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "6a3fd665b34a5979748cff01d8f620ce"
server_key_iv: "a2d22f42b6ae30429659271b2edc8548"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.295665 7567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:49.310465 7567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:49.313669 7567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:49.313869 7567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:49.315500 7567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:49.315578 7567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:49.315651 7567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:49.315702 7567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:35:49 dist-test-slave-06lh krb5kdc[7310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:49.326365 7567 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:42715
I20260928 14:35:49.326447 7697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:42715 every 8 connection(s)
I20260928 14:35:49.327416 7567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:35:49.329190 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 7567
I20260928 14:35:49.329337 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:35:49.329597 6239 external_mini_cluster.cc:1428] Setting key 4015fc4f996073535ea6d52bf2dc0ae4
I20260928 14:35:49.330803 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.316247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:50573 (local address 127.6.23.254:38123)
0928 14:35:49.316405 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.316410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.317753 (+ 1343us) server_negotiation.cc:407] Connection header received
0928 14:35:49.319163 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.319166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.319273 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.319374 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:49.321095 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.321784 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.323633 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.323854 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.323939 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.326914 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:49.326917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:49.326919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:49.326946 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:49.328617 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.329174 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.329177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.329178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.329254 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.329623 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.329627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.329629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.329832 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:49.329939 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.330502 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.330638 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:35:49.331634 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.317079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38123 (local address 127.6.23.194:50573)
0928 14:35:49.317571 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.317637 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.318922 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.319604 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.319615 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.320150 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:49.320917 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.320940 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.321937 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.321942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.323483 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.323497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.324011 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.324017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.324267 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.324985 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:49.325001 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:49.326727 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:49.328722 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.328726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.328737 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.329062 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.329369 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.329372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.329375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.329518 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.329936 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:49.329943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:49.330241 (+ 298us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.330462 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.330776 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260928 14:35:49.334983 7698 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38123
I20260928 14:35:49.335139 7698 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:49.335644 7698 heartbeater.cc:511] Master 127.6.23.254:38123 requested a full tablet report, sending...
I20260928 14:35:49.336669 7354 ts_manager.cc:194] Registered new tserver with Master: 235a96d3782c49e1a85e49a98895dbbd (127.6.23.194:42715)
I20260928 14:35:49.337267 7354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:50573
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:35:49.342768 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.335211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:41869 (local address 127.6.23.254:38123)
0928 14:35:49.335371 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.335375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.335645 (+ 270us) server_negotiation.cc:407] Connection header received
0928 14:35:49.335766 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.335769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.335815 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.335898 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:49.336345 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.337224 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.338226 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.338452 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.338582 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.339740 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:49.339743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:49.339746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:49.339769 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:49.341154 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.341567 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.341571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.341572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.341651 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.341925 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.341930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.341932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.342145 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:49.342254 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.342505 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.342626 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:35:49.342778 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.335159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38123 (local address 127.6.23.194:41869)
0928 14:35:49.335564 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.335575 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.335683 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.335913 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.335916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.336105 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:49.336221 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.336228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.337501 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.337505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.338101 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.338110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.338616 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.338620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.338726 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.339267 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:49.339283 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:49.339599 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:49.341261 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.341266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.341268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.341462 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.341747 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.341749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.341751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.341836 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.342243 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:49.342246 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:49.342345 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.342455 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.342600 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":143,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:35:49.359089 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38123
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:49.470170 7707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:49.470476 7707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:49.470561 7707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:49.474694 7707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:49.474782 7707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:49.474850 7707 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:35:49.474936 7707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:35:49.480304 7707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:38123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:49.481388 7707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:49.482436 7707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:49.491957 7715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:49.491995 7713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:49.492067 7712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:49.492276 7707 server_base.cc:1034] running on GCE node
I20260928 14:35:49.492761 7707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:49.493463 7707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:49.494668 7707 hybrid_clock.cc:648] HybridClock initialized: now 1790606149494648 us; error 31 us; skew 500 ppm
Sep 28 14:35:49 dist-test-slave-06lh krb5kdc[7310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:49.497329 7707 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:35:49.498216 7707 webserver.cc:460] Webserver started at http://127.6.23.195:35169/ using document root <none> and password file <none>
I20260928 14:35:49.498854 7707 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:49.498946 7707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:49.499161 7707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:49.501264 7707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "56e08ed1456345f2b4b70bb82927f8a6"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "68e27c343669d6b8336a5dbd87d9310c"
server_key_iv: "d21fb1156fae63cb57c649919cf30dd7"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.501842 7707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "56e08ed1456345f2b4b70bb82927f8a6"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "68e27c343669d6b8336a5dbd87d9310c"
server_key_iv: "d21fb1156fae63cb57c649919cf30dd7"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.505591 7707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:35:49.508069 7722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:49.509220 7707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:35:49.509366 7707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "56e08ed1456345f2b4b70bb82927f8a6"
format_stamp: "Formatted at 2026-09-28 14:35:49 on dist-test-slave-06lh"
server_key: "68e27c343669d6b8336a5dbd87d9310c"
server_key_iv: "d21fb1156fae63cb57c649919cf30dd7"
server_key_version: "encryptionkey@0"
I20260928 14:35:49.509480 7707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:49.530692 7707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:49.533776 7707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:49.534009 7707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:49.535669 7707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:49.535743 7707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:49.535813 7707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:49.535868 7707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:49.546691 7707 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:41477
I20260928 14:35:49.546887 7837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:41477 every 8 connection(s)
Sep 28 14:35:49 dist-test-slave-06lh krb5kdc[7310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:49.547758 7707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:35:49.551470 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.536378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:45661 (local address 127.6.23.254:38123)
0928 14:35:49.536539 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.536544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.538069 (+ 1525us) server_negotiation.cc:407] Connection header received
0928 14:35:49.539570 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.539575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.539638 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.539728 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:49.541533 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.542447 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.544023 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.544392 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.544532 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.547830 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:49.547834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:49.547837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:49.547869 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:49.549505 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.550038 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.550041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.550042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.550125 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.550429 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.550432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.550433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.550615 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:49.550688 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.551176 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.551346 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:35:49.552340 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.537404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38123 (local address 127.6.23.195:45661)
0928 14:35:49.537903 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.537954 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.539175 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.540049 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.540062 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.540607 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:49.541318 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.541338 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.542616 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.542620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.543877 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.543887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.544739 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.544747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.544999 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.545742 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:49.545762 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:49.547644 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:49.549638 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.549643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.549655 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.549915 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.550211 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.550213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.550214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.550343 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.550702 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:49.550707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:49.550936 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.551108 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.551447 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260928 14:35:49.553723 7838 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38123
I20260928 14:35:49.553910 7838 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:49.554409 7838 heartbeater.cc:511] Master 127.6.23.254:38123 requested a full tablet report, sending...
I20260928 14:35:49.555653 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 7707
I20260928 14:35:49.555673 7354 ts_manager.cc:194] Registered new tserver with Master: 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477)
I20260928 14:35:49.555775 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:35:49.556085 6239 external_mini_cluster.cc:1428] Setting key 42c8561e1c43fc9219407797adf31b26
I20260928 14:35:49.556471 7354 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:45661
I20260928 14:35:49.558192 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:35:49.565552 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.556259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:39707 (local address 127.6.23.254:38123)
0928 14:35:49.556812 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.556818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.556831 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:35:49.556865 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.556868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.556908 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.556997 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:49.557640 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.559682 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.560831 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.561042 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.561127 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.562280 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:49.562284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:49.562287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:49.562338 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:49.563867 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.564295 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.564302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.564306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.564400 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.564662 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.564668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.564673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.564915 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:49.565021 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.565255 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.565396 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463}
I20260928 14:35:49.565587 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.556160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38123 (local address 127.6.23.195:39707)
0928 14:35:49.556464 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.556480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.556587 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.557113 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.557116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.557367 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:49.557522 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.557530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.559813 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.559815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.560699 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.560713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.561168 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.561172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.561273 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.561822 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:49.561834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:49.562144 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:49.563980 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.563983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.563984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.564191 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.564496 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:49.564498 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:49.564499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:49.564574 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:49.565007 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:49.565009 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:49.565091 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.565169 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.565337 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
Sep 28 14:35:49 dist-test-slave-06lh krb5kdc[7310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:35:49 dist-test-slave-06lh krb5kdc[7310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:49.585216 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.577304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.6.23.254:38123)
0928 14:35:49.577440 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.577443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.577454 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:35:49.577487 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.577489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.577522 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.577620 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:49.578392 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.578958 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.579844 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.580053 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.580126 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.582320 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:49.582323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:49.582325 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:49.582347 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:49.583591 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.584007 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.584012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.584014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.584094 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:49.584363 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:49.584366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:49.584368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:49.584539 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:49.584688 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.584927 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.585054 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:35:49.587937 7355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:35:49.590869 7355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:49.605406 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.599257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52366 (local address 127.6.23.195:41477)
0928 14:35:49.599752 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.599764 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.600299 (+ 535us) server_negotiation.cc:407] Connection header received
0928 14:35:49.600639 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.600643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.600824 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.600980 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:49.601333 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.602551 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.604106 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.604685 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.604773 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.604935 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.605146 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.605249 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":105,"threads_started":1}
I20260928 14:35:49.605407 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.599065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:41477 (local address 127.0.0.1:52366)
0928 14:35:49.600163 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.600216 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.600479 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.600999 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.601005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.601044 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:49.601210 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.601218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.602668 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.602672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.603965 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.603976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.604810 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.604817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.604940 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.604978 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.605113 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.605267 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":129,"thread_start_us":78,"threads_started":1}
I20260928 14:35:49.605410 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.599818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:42715 (local address 127.0.0.1:54778)
0928 14:35:49.600264 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.600279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.600404 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.601329 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.601332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.601351 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:49.601469 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.601474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.603097 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.603102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.604078 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.604091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.604902 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.604905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.604999 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.605021 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.605105 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.605251 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":38,"thread_start_us":81,"threads_started":1}
I20260928 14:35:49.606485 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.599170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:36147 (local address 127.0.0.1:57318)
0928 14:35:49.600488 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.600501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.600575 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.601208 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.601210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.601225 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:49.601353 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.601360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.602351 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.602356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.603965 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.603976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.606085 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.606088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.606186 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.606207 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.606305 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.606370 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":89,"thread_start_us":116,"threads_started":1}
I20260928 14:35:49.606904 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.600240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.6.23.193:36147)
0928 14:35:49.600764 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.600775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.600791 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:35:49.600848 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.600852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.601084 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.601235 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:49.601479 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.602235 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.604993 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.605808 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.605927 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.606108 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.606670 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.606765 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":86,"threads_started":1}
I20260928 14:35:49.607625 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.599961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.6.23.194:42715)
0928 14:35:49.600353 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.600364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.600386 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:35:49.600617 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.600622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.600824 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.600980 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:49.601603 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.602967 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.604205 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.604775 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.604885 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.605040 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.607421 (+ 2381us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.607504 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":110,"threads_started":1}
I20260928 14:35:49.608860 7772 tablet_service.cc:1467] Processing CreateTablet for tablet 50c0b96d8e5a457fb7a7392285230a7b (DEFAULT_TABLE table=test-table [id=db567b724a284248be084b2b21e04799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:49.609323 7492 tablet_service.cc:1467] Processing CreateTablet for tablet 50c0b96d8e5a457fb7a7392285230a7b (DEFAULT_TABLE table=test-table [id=db567b724a284248be084b2b21e04799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:49.609993 7772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c0b96d8e5a457fb7a7392285230a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:49.610316 7492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c0b96d8e5a457fb7a7392285230a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:49.610080 7632 tablet_service.cc:1467] Processing CreateTablet for tablet 50c0b96d8e5a457fb7a7392285230a7b (DEFAULT_TABLE table=test-table [id=db567b724a284248be084b2b21e04799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:49.611028 7632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c0b96d8e5a457fb7a7392285230a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:49.616575 7861 tablet_bootstrap.cc:492] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Bootstrap starting.
I20260928 14:35:49.618531 7863 tablet_bootstrap.cc:492] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Bootstrap starting.
I20260928 14:35:49.618536 7862 tablet_bootstrap.cc:492] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Bootstrap starting.
I20260928 14:35:49.619374 7861 tablet_bootstrap.cc:654] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:49.620399 7861 log.cc:824] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:49.620651 7863 tablet_bootstrap.cc:654] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:49.620651 7862 tablet_bootstrap.cc:654] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:49.621479 7863 log.cc:824] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:49.621479 7862 log.cc:824] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:49.622555 7861 tablet_bootstrap.cc:492] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: No bootstrap required, opened a new log
I20260928 14:35:49.622777 7861 ts_tablet_manager.cc:1397] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260928 14:35:49.623335 7863 tablet_bootstrap.cc:492] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: No bootstrap required, opened a new log
I20260928 14:35:49.623520 7863 ts_tablet_manager.cc:1397] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260928 14:35:49.623719 7862 tablet_bootstrap.cc:492] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: No bootstrap required, opened a new log
I20260928 14:35:49.623885 7862 ts_tablet_manager.cc:1397] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:35:49.625876 7861 raft_consensus.cc:348] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.626124 7861 raft_consensus.cc:374] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:49.626179 7861 raft_consensus.cc:729] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb81779fc954885bfdde1411fcb202d, State: Initialized, Role: FOLLOWER
I20260928 14:35:49.626636 7861 consensus_queue.cc:260] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.627375 7863 raft_consensus.cc:348] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.627689 7863 raft_consensus.cc:374] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:49.627692 7861 ts_tablet_manager.cc:1428] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 14:35:49.627755 7863 raft_consensus.cc:729] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235a96d3782c49e1a85e49a98895dbbd, State: Initialized, Role: FOLLOWER
I20260928 14:35:49.627758 7862 raft_consensus.cc:348] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.627977 7862 raft_consensus.cc:374] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:49.628024 7862 raft_consensus.cc:729] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e08ed1456345f2b4b70bb82927f8a6, State: Initialized, Role: FOLLOWER
I20260928 14:35:49.628070 7558 heartbeater.cc:503] Master 127.6.23.254:38123 was elected leader, sending a full tablet report...
I20260928 14:35:49.628448 7862 consensus_queue.cc:260] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.629243 7838 heartbeater.cc:503] Master 127.6.23.254:38123 was elected leader, sending a full tablet report...
I20260928 14:35:49.629242 7863 consensus_queue.cc:260] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.629369 7862 ts_tablet_manager.cc:1428] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:35:49.629979 7863 ts_tablet_manager.cc:1428] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:35:49.630415 7698 heartbeater.cc:503] Master 127.6.23.254:38123 was elected leader, sending a full tablet report...
W20260928 14:35:49.798929 7839 tablet.cc:2369] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:35:49.829324 7699 tablet.cc:2369] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:35:49.861630 7559 tablet.cc:2369] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:35:49.945281 7869 raft_consensus.cc:482] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:49.945567 7869 raft_consensus.cc:504] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.946728 7869 leader_election.cc:288] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477), ccb81779fc954885bfdde1411fcb202d (127.6.23.193:36147)
I20260928 14:35:49.952257 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.947130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:44587 (local address 127.6.23.193:36147)
0928 14:35:49.947314 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.947318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.947565 (+ 247us) server_negotiation.cc:407] Connection header received
0928 14:35:49.947721 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.947724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.947778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.947868 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:49.948254 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.949053 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.950837 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.951480 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.951618 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.951669 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.952045 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.952123 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:35:49.952257 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.947192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:37787 (local address 127.6.23.195:41477)
0928 14:35:49.947424 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:49.947429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:49.947558 (+ 129us) server_negotiation.cc:407] Connection header received
0928 14:35:49.947721 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:49.947724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:49.947778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:49.947875 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:49.948254 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.949058 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.950837 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.951480 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.951616 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.951669 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:49.952055 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 14:35:49.952123 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 14:35:49.952343 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.947019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:36147 (local address 127.6.23.194:44587)
0928 14:35:49.947470 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.947484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.947600 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.947906 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.947909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.947935 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:49.948122 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.948132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.949177 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.949180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.950676 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.950688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.951614 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.951618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.951863 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.951935 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.952024 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.952151 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":468,"spinlock_wait_cycles":17664,"thread_start_us":122,"threads_started":1}
I20260928 14:35:49.952541 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:49.947019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:41477 (local address 127.6.23.194:37787)
0928 14:35:49.947421 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:49.947436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:49.947600 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:49.947906 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:49.947909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:49.947935 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:49.948132 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.948138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.949177 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.949180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.950677 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:49.950688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:49.951614 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:49.951618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:49.951864 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:49.951937 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 14:35:49.952016 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:35:49.952151 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":290,"spinlock_wait_cycles":22272,"thread_start_us":95,"threads_started":1}
I20260928 14:35:49.953089 7512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c0b96d8e5a457fb7a7392285230a7b" candidate_uuid: "235a96d3782c49e1a85e49a98895dbbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb81779fc954885bfdde1411fcb202d" is_pre_election: true
I20260928 14:35:49.953176 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c0b96d8e5a457fb7a7392285230a7b" candidate_uuid: "235a96d3782c49e1a85e49a98895dbbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e08ed1456345f2b4b70bb82927f8a6" is_pre_election: true
I20260928 14:35:49.953456 7792 raft_consensus.cc:2454] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235a96d3782c49e1a85e49a98895dbbd in term 0.
I20260928 14:35:49.953456 7512 raft_consensus.cc:2454] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235a96d3782c49e1a85e49a98895dbbd in term 0.
I20260928 14:35:49.953948 7584 leader_election.cc:302] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235a96d3782c49e1a85e49a98895dbbd, ccb81779fc954885bfdde1411fcb202d; no voters:
I20260928 14:35:49.954190 7869 raft_consensus.cc:2781] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:49.954313 7869 raft_consensus.cc:482] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:49.954394 7869 raft_consensus.cc:3037] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:49.955672 7869 raft_consensus.cc:504] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.956087 7869 leader_election.cc:288] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [CANDIDATE]: Term 1 election: Requested vote from peers 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477), ccb81779fc954885bfdde1411fcb202d (127.6.23.193:36147)
I20260928 14:35:49.956441 7512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c0b96d8e5a457fb7a7392285230a7b" candidate_uuid: "235a96d3782c49e1a85e49a98895dbbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb81779fc954885bfdde1411fcb202d"
I20260928 14:35:49.956436 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c0b96d8e5a457fb7a7392285230a7b" candidate_uuid: "235a96d3782c49e1a85e49a98895dbbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e08ed1456345f2b4b70bb82927f8a6"
I20260928 14:35:49.956568 7512 raft_consensus.cc:3037] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:49.956578 7792 raft_consensus.cc:3037] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:49.957935 7512 raft_consensus.cc:2454] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235a96d3782c49e1a85e49a98895dbbd in term 1.
I20260928 14:35:49.958283 7584 leader_election.cc:302] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235a96d3782c49e1a85e49a98895dbbd, ccb81779fc954885bfdde1411fcb202d; no voters:
I20260928 14:35:49.958412 7792 raft_consensus.cc:2454] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235a96d3782c49e1a85e49a98895dbbd in term 1.
I20260928 14:35:49.958488 7869 raft_consensus.cc:2781] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:49.958691 7869 raft_consensus.cc:686] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 1 LEADER]: Becoming Leader. State: Replica: 235a96d3782c49e1a85e49a98895dbbd, State: Running, Role: LEADER
I20260928 14:35:49.959057 7869 consensus_queue.cc:237] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } }
I20260928 14:35:49.962206 7354 catalog_manager.cc:5495] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 235a96d3782c49e1a85e49a98895dbbd (127.6.23.194). New cstate: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } health_report { overall_health: HEALTHY } } }
I20260928 14:35:50.061465 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.057854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.6.23.194:42715)
0928 14:35:50.057986 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.057990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.058042 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:35:50.058230 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.058232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.058290 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.058378 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:50.058627 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.059142 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.060031 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.060237 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.060314 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.060672 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:50.061086 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:50.061230 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.061300 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.061352 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:35:50.067301 7792 raft_consensus.cc:1261] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Refusing update from remote peer 235a96d3782c49e1a85e49a98895dbbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:50.067440 7512 raft_consensus.cc:1261] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Refusing update from remote peer 235a96d3782c49e1a85e49a98895dbbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:50.068076 7872 consensus_queue.cc:1035] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:50.068243 7869 consensus_queue.cc:1035] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:50.074791 7875 mvcc.cc:205] Tried to move back new op lower bound from 7334322790668537856 to 7334322790235389952. Current Snapshot: MvccSnapshot[applied={T|T < 7334322790668537856}]
I20260928 14:35:50.089366 7355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:50.089560 7355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:50.092151 7355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7: Sending DeleteTablet for 3 replicas of tablet 50c0b96d8e5a457fb7a7392285230a7b
I20260928 14:35:50.092902 7772 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c0b96d8e5a457fb7a7392285230a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:50 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:52366
I20260928 14:35:50.093071 7492 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c0b96d8e5a457fb7a7392285230a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:50 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:57318
I20260928 14:35:50.093423 7632 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c0b96d8e5a457fb7a7392285230a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:50 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:54778
I20260928 14:35:50.093590 7886 tablet_replica.cc:330] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: stopping tablet replica
I20260928 14:35:50.093854 7887 tablet_replica.cc:330] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: stopping tablet replica
I20260928 14:35:50.093927 7886 raft_consensus.cc:2229] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:50.094238 7887 raft_consensus.cc:2229] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:35:50.094252 7886 raft_consensus.cc:2258] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:50.094697 7887 raft_consensus.cc:2258] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:50.094856 7355 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:35:50.095328 7355 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:50.095892 7886 ts_tablet_manager.cc:1905] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:50.096385 7887 ts_tablet_manager.cc:1905] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:50.098769 7886 ts_tablet_manager.cc:1918] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:35:50.098791 7885 tablet_replica.cc:330] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: stopping tablet replica
I20260928 14:35:50.098912 7886 log.cc:1196] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/50c0b96d8e5a457fb7a7392285230a7b
I20260928 14:35:50.099257 7887 ts_tablet_manager.cc:1918] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:35:50.099361 7886 ts_tablet_manager.cc:1939] T 50c0b96d8e5a457fb7a7392285230a7b P ccb81779fc954885bfdde1411fcb202d: Deleting consensus metadata
I20260928 14:35:50.099457 7887 log.cc:1196] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/50c0b96d8e5a457fb7a7392285230a7b
I20260928 14:35:50.099797 7887 ts_tablet_manager.cc:1939] T 50c0b96d8e5a457fb7a7392285230a7b P 235a96d3782c49e1a85e49a98895dbbd: Deleting consensus metadata
I20260928 14:35:50.100780 7492 tablet_service.cc:1467] Processing CreateTablet for tablet cd4f712a5ed542fe9ecdb6d9dbd04119 (DEFAULT_TABLE table=test-table [id=6b3f086472e74c8d8664adf6b2d1bf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:50.100878 7340 catalog_manager.cc:4841] TS 235a96d3782c49e1a85e49a98895dbbd (127.6.23.194:42715): tablet 50c0b96d8e5a457fb7a7392285230a7b (table test-table [id=db567b724a284248be084b2b21e04799]) successfully deleted
I20260928 14:35:50.101070 7492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4f712a5ed542fe9ecdb6d9dbd04119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:50.102715 7861 tablet_bootstrap.cc:492] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: Bootstrap starting.
I20260928 14:35:50.102851 7632 tablet_service.cc:1467] Processing CreateTablet for tablet cd4f712a5ed542fe9ecdb6d9dbd04119 (DEFAULT_TABLE table=test-table [id=6b3f086472e74c8d8664adf6b2d1bf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:50.103132 7632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4f712a5ed542fe9ecdb6d9dbd04119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:50.103710 7861 tablet_bootstrap.cc:654] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:50.104146 7885 raft_consensus.cc:2229] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:50.104265 7340 catalog_manager.cc:4841] TS ccb81779fc954885bfdde1411fcb202d (127.6.23.193:36147): tablet 50c0b96d8e5a457fb7a7392285230a7b (table test-table [id=db567b724a284248be084b2b21e04799]) successfully deleted
I20260928 14:35:50.104631 7885 raft_consensus.cc:2258] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:50.105370 7863 tablet_bootstrap.cc:492] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: Bootstrap starting.
I20260928 14:35:50.106208 7772 tablet_service.cc:1467] Processing CreateTablet for tablet cd4f712a5ed542fe9ecdb6d9dbd04119 (DEFAULT_TABLE table=test-table [id=6b3f086472e74c8d8664adf6b2d1bf3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:50.106352 7863 tablet_bootstrap.cc:654] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:50.106503 7772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4f712a5ed542fe9ecdb6d9dbd04119. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:50.107653 7863 tablet_bootstrap.cc:492] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: No bootstrap required, opened a new log
I20260928 14:35:50.107750 7863 ts_tablet_manager.cc:1397] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260928 14:35:50.108245 7863 raft_consensus.cc:348] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.108361 7863 raft_consensus.cc:374] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:50.108397 7863 raft_consensus.cc:729] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235a96d3782c49e1a85e49a98895dbbd, State: Initialized, Role: FOLLOWER
I20260928 14:35:50.108527 7863 consensus_queue.cc:260] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.108727 7885 ts_tablet_manager.cc:1905] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:50.108904 7863 ts_tablet_manager.cc:1428] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:50.109349 7862 tablet_bootstrap.cc:492] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: Bootstrap starting.
I20260928 14:35:50.110983 7862 tablet_bootstrap.cc:654] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:50.111891 7885 ts_tablet_manager.cc:1918] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:35:50.111981 7885 log.cc:1196] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/50c0b96d8e5a457fb7a7392285230a7b
I20260928 14:35:50.112080 7861 tablet_bootstrap.cc:492] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: No bootstrap required, opened a new log
I20260928 14:35:50.112169 7861 ts_tablet_manager.cc:1397] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 14:35:50.112288 7885 ts_tablet_manager.cc:1939] T 50c0b96d8e5a457fb7a7392285230a7b P 56e08ed1456345f2b4b70bb82927f8a6: Deleting consensus metadata
I20260928 14:35:50.112689 7861 raft_consensus.cc:348] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.112808 7861 raft_consensus.cc:374] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:50.112840 7861 raft_consensus.cc:729] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb81779fc954885bfdde1411fcb202d, State: Initialized, Role: FOLLOWER
I20260928 14:35:50.112957 7861 consensus_queue.cc:260] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.113257 7861 ts_tablet_manager.cc:1428] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:50.113479 7342 catalog_manager.cc:4841] TS 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477): tablet 50c0b96d8e5a457fb7a7392285230a7b (table test-table [id=db567b724a284248be084b2b21e04799]) successfully deleted
I20260928 14:35:50.114393 7862 tablet_bootstrap.cc:492] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: No bootstrap required, opened a new log
I20260928 14:35:50.114465 7862 ts_tablet_manager.cc:1397] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 14:35:50.114826 7862 raft_consensus.cc:348] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.114921 7862 raft_consensus.cc:374] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:50.114948 7862 raft_consensus.cc:729] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e08ed1456345f2b4b70bb82927f8a6, State: Initialized, Role: FOLLOWER
I20260928 14:35:50.115058 7862 consensus_queue.cc:260] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.116632 7862 ts_tablet_manager.cc:1428] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:35:50.484208 7867 raft_consensus.cc:482] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:50.484345 7867 raft_consensus.cc:504] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.485493 7867 leader_election.cc:288] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 235a96d3782c49e1a85e49a98895dbbd (127.6.23.194:42715), 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477)
I20260928 14:35:50.490834 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.485892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:45667 (local address 127.6.23.194:42715)
0928 14:35:50.486028 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.486034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.486341 (+ 307us) server_negotiation.cc:407] Connection header received
0928 14:35:50.486507 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.486510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.486562 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.486658 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:50.487019 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.487843 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.489672 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.490292 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.490414 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.490471 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.490617 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.490692 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:35:50.490890 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.485752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:42715 (local address 127.6.23.193:45667)
0928 14:35:50.486230 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:50.486246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:50.486383 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:50.486681 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:50.486684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:50.486727 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:50.486901 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.486909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.487963 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.487966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.489525 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.489536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.490408 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.490410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.490478 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.490499 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:35:50.490601 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:35:50.490677 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":229,"spinlock_wait_cycles":13184,"thread_start_us":92,"threads_started":1}
I20260928 14:35:50.491106 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.485752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:41477 (local address 127.6.23.193:55215)
0928 14:35:50.486312 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:50.486325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:50.486394 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:50.486681 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:50.486684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:50.486725 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:50.486907 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.486912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.487963 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.487966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.489524 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.489536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.490614 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.490617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.490697 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.490718 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:35:50.490791 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:35:50.490848 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":385,"spinlock_wait_cycles":29312,"thread_start_us":85,"threads_started":1}
I20260928 14:35:50.491422 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4f712a5ed542fe9ecdb6d9dbd04119" candidate_uuid: "ccb81779fc954885bfdde1411fcb202d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235a96d3782c49e1a85e49a98895dbbd" is_pre_election: true
I20260928 14:35:50.491509 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.485857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:55215 (local address 127.6.23.195:41477)
0928 14:35:50.486092 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.486097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.486392 (+ 295us) server_negotiation.cc:407] Connection header received
0928 14:35:50.486507 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.486510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.486562 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.486659 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:50.487018 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.487843 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.489852 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.490486 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.490631 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.490676 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.490823 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.491374 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":82,"threads_started":1}
I20260928 14:35:50.491680 7652 raft_consensus.cc:2454] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb81779fc954885bfdde1411fcb202d in term 0.
I20260928 14:35:50.491935 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4f712a5ed542fe9ecdb6d9dbd04119" candidate_uuid: "ccb81779fc954885bfdde1411fcb202d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e08ed1456345f2b4b70bb82927f8a6" is_pre_election: true
I20260928 14:35:50.492091 7792 raft_consensus.cc:2454] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb81779fc954885bfdde1411fcb202d in term 0.
I20260928 14:35:50.492192 7444 leader_election.cc:302] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235a96d3782c49e1a85e49a98895dbbd, ccb81779fc954885bfdde1411fcb202d; no voters:
I20260928 14:35:50.492488 7867 raft_consensus.cc:2781] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:50.492544 7867 raft_consensus.cc:482] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:50.492594 7867 raft_consensus.cc:3037] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:50.493736 7867 raft_consensus.cc:504] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.494094 7867 leader_election.cc:288] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [CANDIDATE]: Term 1 election: Requested vote from peers 235a96d3782c49e1a85e49a98895dbbd (127.6.23.194:42715), 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477)
I20260928 14:35:50.494406 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4f712a5ed542fe9ecdb6d9dbd04119" candidate_uuid: "ccb81779fc954885bfdde1411fcb202d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e08ed1456345f2b4b70bb82927f8a6"
I20260928 14:35:50.494422 7652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4f712a5ed542fe9ecdb6d9dbd04119" candidate_uuid: "ccb81779fc954885bfdde1411fcb202d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "235a96d3782c49e1a85e49a98895dbbd"
I20260928 14:35:50.494544 7792 raft_consensus.cc:3037] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:50.494560 7652 raft_consensus.cc:3037] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:50.495452 7792 raft_consensus.cc:2454] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb81779fc954885bfdde1411fcb202d in term 1.
I20260928 14:35:50.495630 7652 raft_consensus.cc:2454] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb81779fc954885bfdde1411fcb202d in term 1.
I20260928 14:35:50.495739 7446 leader_election.cc:302] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e08ed1456345f2b4b70bb82927f8a6, ccb81779fc954885bfdde1411fcb202d; no voters:
I20260928 14:35:50.495960 7867 raft_consensus.cc:2781] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:50.496206 7867 raft_consensus.cc:686] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 1 LEADER]: Becoming Leader. State: Replica: ccb81779fc954885bfdde1411fcb202d, State: Running, Role: LEADER
I20260928 14:35:50.496667 7867 consensus_queue.cc:237] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.498955 7354 catalog_manager.cc:5495] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb81779fc954885bfdde1411fcb202d (127.6.23.193). New cstate: current_term: 1 leader_uuid: "ccb81779fc954885bfdde1411fcb202d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:35:50 dist-test-slave-06lh krb5kdc[7310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606148, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:50.597478 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.589061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38123 (local address 127.0.0.1:52318)
0928 14:35:50.589371 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:50.589384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:50.589481 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:50.589714 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:50.589716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:50.590113 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:50.590207 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.590213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.591001 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.591003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.591671 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.591681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.592123 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.592126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.592186 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.592746 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:50.592757 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:50.594355 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:50.596065 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:50.596074 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:50.596080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:50.596309 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:50.596556 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:50.596558 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:50.596559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:50.596633 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:50.596966 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:50.596971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:50.597079 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 14:35:50.597170 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:35:50.597294 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":127,"threads_started":1}
I20260928 14:35:50.597486 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.589095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.6.23.254:38123)
0928 14:35:50.589494 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.589501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.589517 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:35:50.589556 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.589558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.589599 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.589682 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:50.590363 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.590906 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.591778 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.592025 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.592103 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.594484 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:50.594487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:50.594489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:50.594521 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:50.595959 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:50.596393 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:50.596395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:50.596396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:50.596470 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:50.596707 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:50.596709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:50.596710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:50.596878 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:50.596953 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.597230 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.597355 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":337,"thread_start_us":107,"threads_started":1}
I20260928 14:35:50.601486 7354 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:52318:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:35:50.602209 7354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:50.606953 7632 tablet_service.cc:1467] Processing CreateTablet for tablet fa8a427550d444f692f51b96715340a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03cd7d873a3f459f88216013c41d726e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:35:50.606953 7492 tablet_service.cc:1467] Processing CreateTablet for tablet fa8a427550d444f692f51b96715340a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03cd7d873a3f459f88216013c41d726e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:35:50.607107 7772 tablet_service.cc:1467] Processing CreateTablet for tablet fa8a427550d444f692f51b96715340a8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=03cd7d873a3f459f88216013c41d726e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:35:50.607393 7492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8a427550d444f692f51b96715340a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:50.607393 7632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8a427550d444f692f51b96715340a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:50.607515 7772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8a427550d444f692f51b96715340a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:50.609936 7862 tablet_bootstrap.cc:492] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: Bootstrap starting.
I20260928 14:35:50.610194 7861 tablet_bootstrap.cc:492] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: Bootstrap starting.
I20260928 14:35:50.610330 7903 tablet_bootstrap.cc:492] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: Bootstrap starting.
I20260928 14:35:50.610906 7862 tablet_bootstrap.cc:654] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:50.611121 7861 tablet_bootstrap.cc:654] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:50.611182 7903 tablet_bootstrap.cc:654] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:50.612221 7862 tablet_bootstrap.cc:492] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: No bootstrap required, opened a new log
I20260928 14:35:50.612282 7861 tablet_bootstrap.cc:492] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: No bootstrap required, opened a new log
I20260928 14:35:50.612316 7862 ts_tablet_manager.cc:1397] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:35:50.612347 7861 ts_tablet_manager.cc:1397] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:50.612470 7903 tablet_bootstrap.cc:492] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: No bootstrap required, opened a new log
I20260928 14:35:50.612574 7903 ts_tablet_manager.cc:1397] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:50.612757 7862 raft_consensus.cc:348] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.612757 7861 raft_consensus.cc:348] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.612864 7862 raft_consensus.cc:374] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:50.612864 7861 raft_consensus.cc:374] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:50.612897 7862 raft_consensus.cc:729] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e08ed1456345f2b4b70bb82927f8a6, State: Initialized, Role: FOLLOWER
I20260928 14:35:50.612897 7861 raft_consensus.cc:729] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb81779fc954885bfdde1411fcb202d, State: Initialized, Role: FOLLOWER
I20260928 14:35:50.613030 7862 consensus_queue.cc:260] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.613036 7861 consensus_queue.cc:260] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.613075 7903 raft_consensus.cc:348] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.613196 7903 raft_consensus.cc:374] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:50.613278 7903 raft_consensus.cc:729] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 235a96d3782c49e1a85e49a98895dbbd, State: Initialized, Role: FOLLOWER
I20260928 14:35:50.613282 7862 ts_tablet_manager.cc:1428] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:35:50.613282 7861 ts_tablet_manager.cc:1428] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:50.613507 7868 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.613490 7867 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.613631 7868 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:50.613631 7867 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:50.613597 7903 consensus_queue.cc:260] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.613895 7903 ts_tablet_manager.cc:1428] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:50.614606 7907 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.614754 7907 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:50.630779 7907 raft_consensus.cc:482] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:50.630944 7907 raft_consensus.cc:504] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.631397 7907 leader_election.cc:288] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb81779fc954885bfdde1411fcb202d (127.6.23.193:36147), 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477)
I20260928 14:35:50.631831 7512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8a427550d444f692f51b96715340a8" candidate_uuid: "235a96d3782c49e1a85e49a98895dbbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb81779fc954885bfdde1411fcb202d" is_pre_election: true
I20260928 14:35:50.631852 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8a427550d444f692f51b96715340a8" candidate_uuid: "235a96d3782c49e1a85e49a98895dbbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e08ed1456345f2b4b70bb82927f8a6" is_pre_election: true
I20260928 14:35:50.631969 7512 raft_consensus.cc:2454] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235a96d3782c49e1a85e49a98895dbbd in term 0.
I20260928 14:35:50.632001 7792 raft_consensus.cc:2454] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 235a96d3782c49e1a85e49a98895dbbd in term 0.
I20260928 14:35:50.632283 7584 leader_election.cc:302] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235a96d3782c49e1a85e49a98895dbbd, ccb81779fc954885bfdde1411fcb202d; no voters:
I20260928 14:35:50.632467 7907 raft_consensus.cc:2781] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:50.632547 7907 raft_consensus.cc:482] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:50.632601 7907 raft_consensus.cc:3037] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:50.633879 7907 raft_consensus.cc:504] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.634238 7907 leader_election.cc:288] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [CANDIDATE]: Term 1 election: Requested vote from peers ccb81779fc954885bfdde1411fcb202d (127.6.23.193:36147), 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477)
I20260928 14:35:50.634603 7792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8a427550d444f692f51b96715340a8" candidate_uuid: "235a96d3782c49e1a85e49a98895dbbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e08ed1456345f2b4b70bb82927f8a6"
I20260928 14:35:50.634603 7512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8a427550d444f692f51b96715340a8" candidate_uuid: "235a96d3782c49e1a85e49a98895dbbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb81779fc954885bfdde1411fcb202d"
I20260928 14:35:50.634719 7792 raft_consensus.cc:3037] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:50.634719 7512 raft_consensus.cc:3037] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:50.635787 7512 raft_consensus.cc:2454] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235a96d3782c49e1a85e49a98895dbbd in term 1.
I20260928 14:35:50.635787 7792 raft_consensus.cc:2454] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 235a96d3782c49e1a85e49a98895dbbd in term 1.
I20260928 14:35:50.636083 7586 leader_election.cc:302] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 235a96d3782c49e1a85e49a98895dbbd, 56e08ed1456345f2b4b70bb82927f8a6; no voters:
I20260928 14:35:50.636286 7907 raft_consensus.cc:2781] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:50.636456 7907 raft_consensus.cc:686] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [term 1 LEADER]: Becoming Leader. State: Replica: 235a96d3782c49e1a85e49a98895dbbd, State: Running, Role: LEADER
I20260928 14:35:50.636612 7907 consensus_queue.cc:237] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } }
I20260928 14:35:50.637333 7908 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: TxnStatusTablet state changed. Reason: New leader 235a96d3782c49e1a85e49a98895dbbd. Latest consensus state: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.637486 7908 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:35:50.637871 7910 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:35:50.638063 7910 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:35:50.638363 7353 catalog_manager.cc:5495] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 235a96d3782c49e1a85e49a98895dbbd (127.6.23.194). New cstate: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } health_report { overall_health: UNKNOWN } } }
I20260928 14:35:50.657583 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.654069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.6.23.194:42715)
0928 14:35:50.654207 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.654212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.654226 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:35:50.654395 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.654401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.654478 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.654578 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:50.654834 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.655539 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.656462 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.656696 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.656807 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.657094 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:50.657175 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:50.657258 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.657385 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.657436 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:35:50.657583 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.653948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:42715 (local address 127.0.0.1:54796)
0928 14:35:50.654143 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:50.654160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:50.654275 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:50.654599 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:50.654602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:50.654613 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:50.654725 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.654730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.655654 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.655659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.656342 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.656351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.656808 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.656812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.656903 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.656948 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:50.657264 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:50.657283 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:35:50.657371 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:35:50.657456 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 14:35:50.660708 7512 raft_consensus.cc:1261] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Refusing update from remote peer 235a96d3782c49e1a85e49a98895dbbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:50.660708 7792 raft_consensus.cc:1261] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Refusing update from remote peer 235a96d3782c49e1a85e49a98895dbbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:50.661146 7907 consensus_queue.cc:1035] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:50.661324 7908 consensus_queue.cc:1035] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:50.662719 7867 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: TxnStatusTablet state changed. Reason: New leader 235a96d3782c49e1a85e49a98895dbbd. Latest consensus state: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.662839 7867 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:50.662811 7868 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: TxnStatusTablet state changed. Reason: New leader 235a96d3782c49e1a85e49a98895dbbd. Latest consensus state: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.662931 7868 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:50.664978 7907 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.665091 7907 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:35:50.665201 7907 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.665262 7907 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P 235a96d3782c49e1a85e49a98895dbbd: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:35:50.666507 7915 mvcc.cc:205] Tried to move back new op lower bound from 7334322793101783040 to 7334322793009184768. Current Snapshot: MvccSnapshot[applied={T|T < 7334322793101783040}]
I20260928 14:35:50.666694 7868 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.666797 7868 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P 56e08ed1456345f2b4b70bb82927f8a6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:50.667150 7867 tablet_replica.cc:439] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "235a96d3782c49e1a85e49a98895dbbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb81779fc954885bfdde1411fcb202d" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 36147 } } peers { permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 } } peers { permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 } } }
I20260928 14:35:50.667287 7867 tablet_replica.cc:442] T fa8a427550d444f692f51b96715340a8 P ccb81779fc954885bfdde1411fcb202d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:50.670168 7914 mvcc.cc:205] Tried to move back new op lower bound from 7334322793101783040 to 7334322793009184768. Current Snapshot: MvccSnapshot[applied={T|T < 7334322793101783040 or (T in {7334322793101783040})}]
I20260928 14:35:50.675649 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.671497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.6.23.193:36147)
0928 14:35:50.671741 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.671748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.671760 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:35:50.671902 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.671904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.671952 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.672055 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:50.672306 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.672857 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.674081 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.674320 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.674394 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.674712 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:50.675181 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:50.675352 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.675431 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.675478 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260928 14:35:50.696717 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.692770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:53221 (local address 127.6.23.194:42715)
0928 14:35:50.692926 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.692931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.692946 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:35:50.693001 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.693004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.693056 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.693129 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:50.693519 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.694338 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.695669 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.695893 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.695979 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.696311 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:50.696382 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:50.696434 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.696559 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.696606 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:35:50.697045 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.691270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:42715 (local address 127.6.23.193:53221)
0928 14:35:50.691873 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:50.691900 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:50.692014 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:50.693246 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:50.693249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:50.693259 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:50.693396 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.693403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.694485 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.694489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.695542 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.695553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.696012 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.696015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.696116 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.696176 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:50.696464 (+ 288us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:50.696471 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:35:50.696539 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:35:50.696620 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":134,"threads_started":1}
I20260928 14:35:50.705070 7792 raft_consensus.cc:1261] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Refusing update from remote peer ccb81779fc954885bfdde1411fcb202d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:50.705188 7652 raft_consensus.cc:1261] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Refusing update from remote peer ccb81779fc954885bfdde1411fcb202d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:50.705717 7894 consensus_queue.cc:1035] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [LEADER]: Connected to new peer: Peer: permanent_uuid: "235a96d3782c49e1a85e49a98895dbbd" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:50.705914 7867 consensus_queue.cc:1035] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e08ed1456345f2b4b70bb82927f8a6" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 41477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:50.731164 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.727403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:44539 (local address 127.6.23.193:36147)
0928 14:35:50.727531 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.727535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.727796 (+ 261us) server_negotiation.cc:407] Connection header received
0928 14:35:50.727932 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.727934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.727973 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.728033 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:50.728285 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.728946 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.730184 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.730394 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.730476 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.730776 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:50.730840 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:50.730892 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.731017 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.731060 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:35:50.731254 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.727358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:36147 (local address 127.6.23.194:44539)
0928 14:35:50.727703 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:50.727719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:50.727825 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:50.728069 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:50.728072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:50.728079 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:50.728187 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.728193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.729064 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.729067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.730063 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:50.730074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.730506 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.730510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:50.730600 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.730651 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:50.730926 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:50.730932 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:35:50.730997 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:35:50.731079 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":118,"threads_started":1}
I20260928 14:35:50.755914 7353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:50.756106 7353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:50.758298 7353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa6ad0d1edc34018b7674dfb7741e3f7: Sending DeleteTablet for 3 replicas of tablet cd4f712a5ed542fe9ecdb6d9dbd04119
I20260928 14:35:50.759025 7772 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4f712a5ed542fe9ecdb6d9dbd04119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:50 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:52366
I20260928 14:35:50.759372 7492 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4f712a5ed542fe9ecdb6d9dbd04119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:50 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:57318
I20260928 14:35:50.759451 7935 tablet_replica.cc:330] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: stopping tablet replica
I20260928 14:35:50.759574 7935 raft_consensus.cc:2229] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:50.759733 7935 raft_consensus.cc:2258] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:50.760504 7935 ts_tablet_manager.cc:1905] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:50.760821 7632 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4f712a5ed542fe9ecdb6d9dbd04119 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:50 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:54778
I20260928 14:35:50.761224 7938 tablet_replica.cc:330] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: stopping tablet replica
I20260928 14:35:50.761379 7938 raft_consensus.cc:2229] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:50.761526 7938 raft_consensus.cc:2258] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:50.762233 7938 ts_tablet_manager.cc:1905] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:50.763386 7936 tablet_replica.cc:330] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: stopping tablet replica
I20260928 14:35:50.763585 7935 ts_tablet_manager.cc:1918] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:50.763712 7935 log.cc:1196] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/cd4f712a5ed542fe9ecdb6d9dbd04119
I20260928 14:35:50.764053 7935 ts_tablet_manager.cc:1939] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 56e08ed1456345f2b4b70bb82927f8a6: Deleting consensus metadata
I20260928 14:35:50.764804 7342 catalog_manager.cc:4841] TS 56e08ed1456345f2b4b70bb82927f8a6 (127.6.23.195:41477): tablet cd4f712a5ed542fe9ecdb6d9dbd04119 (table test-table [id=6b3f086472e74c8d8664adf6b2d1bf3f]) successfully deleted
I20260928 14:35:50.765161 7938 ts_tablet_manager.cc:1918] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:50.765244 7938 log.cc:1196] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/cd4f712a5ed542fe9ecdb6d9dbd04119
I20260928 14:35:50.765550 7938 ts_tablet_manager.cc:1939] T cd4f712a5ed542fe9ecdb6d9dbd04119 P 235a96d3782c49e1a85e49a98895dbbd: Deleting consensus metadata
I20260928 14:35:50.766201 7340 catalog_manager.cc:4841] TS 235a96d3782c49e1a85e49a98895dbbd (127.6.23.194:42715): tablet cd4f712a5ed542fe9ecdb6d9dbd04119 (table test-table [id=6b3f086472e74c8d8664adf6b2d1bf3f]) successfully deleted
I20260928 14:35:50.767400 7936 raft_consensus.cc:2229] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:35:50.767738 7936 raft_consensus.cc:2258] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:50.769498 7936 ts_tablet_manager.cc:1905] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:50.772258 7936 ts_tablet_manager.cc:1918] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:50.772337 7936 log.cc:1196] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/cd4f712a5ed542fe9ecdb6d9dbd04119
I20260928 14:35:50.772605 7936 ts_tablet_manager.cc:1939] T cd4f712a5ed542fe9ecdb6d9dbd04119 P ccb81779fc954885bfdde1411fcb202d: Deleting consensus metadata
I20260928 14:35:50.773337 7340 catalog_manager.cc:4841] TS ccb81779fc954885bfdde1411fcb202d (127.6.23.193:36147): tablet cd4f712a5ed542fe9ecdb6d9dbd04119 (table test-table [id=6b3f086472e74c8d8664adf6b2d1bf3f]) successfully deleted
Sep 28 14:35:50 dist-test-slave-06lh krb5kdc[7310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606149, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:35:50.787154 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.773576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57332 (local address 127.6.23.193:36147)
0928 14:35:50.773696 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.773701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.775249 (+ 1548us) server_negotiation.cc:407] Connection header received
0928 14:35:50.775359 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.775363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.775442 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.775536 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:50.776357 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.777120 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.778324 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.778678 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.778799 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.782700 (+ 3901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:50.782711 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:50.782729 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:50.782779 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:50.784989 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:50.785504 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:50.785511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:50.785513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:50.785600 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:50.785940 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:50.785944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:50.785946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:50.786255 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:50.786466 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.786698 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.786825 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 14:35:50.787640 7552 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57332
I20260928 14:35:50.800083 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:50.789452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.6.23.254:38123)
0928 14:35:50.789604 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:50.789609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:50.791524 (+ 1915us) server_negotiation.cc:407] Connection header received
0928 14:35:50.791624 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:50.791627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:50.791679 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:50.791775 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:50.792508 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.793385 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.794544 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:50.794905 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:50.795009 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:50.796599 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:50.796603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:50.796606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:50.796636 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:50.798299 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:50.798764 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:50.798768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:50.798770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:50.798863 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:50.799140 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:50.799142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:50.799144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:50.799413 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:50.799598 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:50.799779 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 14:35:50.799907 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 14:35:50.800952 7353 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52324
I20260928 14:35:50.802350 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 7427
I20260928 14:35:50.809689 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 7567
I20260928 14:35:50.815888 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 7707
I20260928 14:35:50.821655 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 7323
2026-09-28T14:35:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3383 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:35:50 dist-test-slave-06lh krb5kdc[7944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:35:50 dist-test-slave-06lh krb5kdc[7944](info): set up 2 sockets
Sep 28 14:35:50 dist-test-slave-06lh krb5kdc[7944](info): commencing operation
krb5kdc: starting...
W20260928 14:35:51.860270 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:35:51 dist-test-slave-06lh krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:35:51.980054 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35779
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:35779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:52.091254 7957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:52.091562 7957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:52.091645 7957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:52.095716 7957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:35:52.095790 7957 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:35:52.095821 7957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:52.095878 7957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:52.095937 7957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:35:52.095973 7957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:35:52.101197 7957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:35779
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:52.102257 7957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:52.103291 7957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:52.111908 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:52.111917 7962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:52.111917 7963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:52.112143 7957 server_base.cc:1034] running on GCE node
I20260928 14:35:52.112756 7957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:52.113961 7957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:52.115156 7957 hybrid_clock.cc:648] HybridClock initialized: now 1790606152115139 us; error 30 us; skew 500 ppm
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:52.118018 7957 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:35:52.118919 7957 webserver.cc:460] Webserver started at http://127.6.23.254:38743/ using document root <none> and password file <none>
I20260928 14:35:52.119563 7957 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:52.119657 7957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:52.119889 7957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:52.121938 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "374a8da70b2c44538b81461cae778c82"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "35158dc29247fde47e38ab58d18130c5"
server_key_iv: "de4d7379ae5d70aa2448f710cd14fcba"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.122531 7957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "374a8da70b2c44538b81461cae778c82"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "35158dc29247fde47e38ab58d18130c5"
server_key_iv: "de4d7379ae5d70aa2448f710cd14fcba"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.126411 7957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 14:35:52.129087 7972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:52.130380 7957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:52.130529 7957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "374a8da70b2c44538b81461cae778c82"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "35158dc29247fde47e38ab58d18130c5"
server_key_iv: "de4d7379ae5d70aa2448f710cd14fcba"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.130653 7957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:52.158738 7957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:52.162317 7957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:52.162536 7957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:52.170615 7957 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:35779
I20260928 14:35:52.170644 8034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:35779 every 8 connection(s)
I20260928 14:35:52.171612 7957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:35:52.174345 8035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:52.176682 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 7957
I20260928 14:35:52.176784 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:35:52.177063 6239 external_mini_cluster.cc:1428] Setting key 1f3fa7e8b86dd7ce54128172fbab1aef
I20260928 14:35:52.180421 8035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: Bootstrap starting.
I20260928 14:35:52.182642 8035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:52.183444 8035 log.cc:824] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:52.185839 8035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: No bootstrap required, opened a new log
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:52.190546 8035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a8da70b2c44538b81461cae778c82" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35779 } }
I20260928 14:35:52.190819 8035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:52.190865 8035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374a8da70b2c44538b81461cae778c82, State: Initialized, Role: FOLLOWER
I20260928 14:35:52.191347 8035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a8da70b2c44538b81461cae778c82" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35779 } }
I20260928 14:35:52.191507 8035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:35:52.191574 8035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:35:52.191605 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.178522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.6.23.254:35779)
0928 14:35:52.178971 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.178985 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.179032 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:35:52.179755 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.179774 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.180126 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.180470 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:52.181270 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.182298 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.183265 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.183810 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.183988 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.186640 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:52.186645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:52.186665 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:52.186700 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:52.188619 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.189027 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.189033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.189039 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.189176 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.189428 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.189431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.189433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.189939 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:52.190147 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.190332 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.190620 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":140,"threads_started":1}
I20260928 14:35:52.191675 8035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:52.192595 8035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a8da70b2c44538b81461cae778c82" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35779 } }
I20260928 14:35:52.192946 8035 leader_election.cc:302] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 374a8da70b2c44538b81461cae778c82; no voters:
I20260928 14:35:52.193272 8035 leader_election.cc:288] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:35:52.193419 8040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:52.193657 8040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [term 1 LEADER]: Becoming Leader. State: Replica: 374a8da70b2c44538b81461cae778c82, State: Running, Role: LEADER
I20260928 14:35:52.194020 8040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a8da70b2c44538b81461cae778c82" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35779 } }
I20260928 14:35:52.194502 8035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:35:52.195590 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 374a8da70b2c44538b81461cae778c82. Latest consensus state: current_term: 1 leader_uuid: "374a8da70b2c44538b81461cae778c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a8da70b2c44538b81461cae778c82" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35779 } } }
I20260928 14:35:52.195715 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:52.196098 8049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:35:52.196079 8041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "374a8da70b2c44538b81461cae778c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "374a8da70b2c44538b81461cae778c82" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35779 } } }
I20260928 14:35:52.196308 8041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82 [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:52.198982 8049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:35:52.204752 8049 catalog_manager.cc:1347] Generated new cluster ID: 310fa1f2563c4bd7a9d478d86983ce55
I20260928 14:35:52.204826 8049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:35:52.217223 8049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:35:52.218602 8049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:35:52.225900 8049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: Generated new TSK 0
I20260928 14:35:52.226718 8049 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:35:52.262223 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35779
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:52.373216 8061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:52.373492 8061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:52.373539 8061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:52.377694 8061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:52.377761 8061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:52.377802 8061 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:35:52.377817 8061 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:35:52.377902 8061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:35:52.383149 8061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:35779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:52.384312 8061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:52.385401 8061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:52.395252 8069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:52.395246 8067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:52.395296 8061 server_base.cc:1034] running on GCE node
W20260928 14:35:52.395249 8066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:52.395987 8061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:52.396652 8061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:52.397832 8061 hybrid_clock.cc:648] HybridClock initialized: now 1790606152397817 us; error 29 us; skew 500 ppm
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:52.400655 8061 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:35:52.401480 8061 webserver.cc:460] Webserver started at http://127.6.23.193:39179/ using document root <none> and password file <none>
I20260928 14:35:52.402029 8061 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:52.402115 8061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:52.402352 8061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:52.404526 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "3a762fc6629d414e8e61f52f6e39f5b3"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "ecb161f99141bac2233a31b9c93b3a02"
server_key_iv: "63c8e5c9811621fcff4c31e14cdbb89f"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.405120 8061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "3a762fc6629d414e8e61f52f6e39f5b3"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "ecb161f99141bac2233a31b9c93b3a02"
server_key_iv: "63c8e5c9811621fcff4c31e14cdbb89f"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.408774 8061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:35:52.411116 8076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:52.412348 8061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:52.412479 8061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "3a762fc6629d414e8e61f52f6e39f5b3"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "ecb161f99141bac2233a31b9c93b3a02"
server_key_iv: "63c8e5c9811621fcff4c31e14cdbb89f"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.412595 8061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:52.441403 8061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:52.444732 8061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:52.445036 8061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:52.446666 8061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:52.446739 8061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:52.446807 8061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:52.446892 8061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:52.458053 8061 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:41103
I20260928 14:35:52.458122 8191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:41103 every 8 connection(s)
I20260928 14:35:52.459276 8061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:35:52.461704 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.447464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:39447 (local address 127.6.23.254:35779)
0928 14:35:52.447595 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.447601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.449406 (+ 1805us) server_negotiation.cc:407] Connection header received
0928 14:35:52.450537 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.450541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.450599 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.450707 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:52.451980 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.452671 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.453940 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.454282 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.454406 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.457592 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:52.457595 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:52.457597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:52.457622 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:52.459459 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.460082 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.460091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.460093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.460170 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.460580 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.460583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.460585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.460780 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:52.460905 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.461396 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.461530 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:35:52.462450 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.448571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35779 (local address 127.6.23.193:39447)
0928 14:35:52.449190 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.449235 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.450305 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.450896 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.450903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.451301 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:52.451806 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.451820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.452794 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.452798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.453805 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.453816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.454461 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.454465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.454634 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.455849 (+ 1215us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:52.455866 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:52.457409 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:52.459569 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.459574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.459586 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.459949 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.460266 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.460271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.460273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.460453 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.460905 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:52.460913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:52.461148 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.461330 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.461669 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":128,"threads_started":1}
I20260928 14:35:52.463706 8192 heartbeater.cc:347] Connected to a master server at 127.6.23.254:35779
I20260928 14:35:52.463869 8192 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:52.464340 8192 heartbeater.cc:511] Master 127.6.23.254:35779 requested a full tablet report, sending...
I20260928 14:35:52.465878 7988 ts_manager.cc:194] Registered new tserver with Master: 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103)
I20260928 14:35:52.467295 7988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:39447
I20260928 14:35:52.469172 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 8061
I20260928 14:35:52.469292 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:35:52.469591 6239 external_mini_cluster.cc:1428] Setting key c69b4bd3bb6b90e809101b93e3111028
I20260928 14:35:52.473093 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.465903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:44249 (local address 127.6.23.254:35779)
0928 14:35:52.466030 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.466033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.466155 (+ 122us) server_negotiation.cc:407] Connection header received
0928 14:35:52.466273 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.466275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.466311 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.466368 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:52.466840 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.467476 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.468490 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.468728 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.468838 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.469954 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:52.469958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:52.469961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:52.469984 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:52.471474 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.471912 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.471915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.471916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.471984 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.472290 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.472296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.472299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.472498 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:52.472579 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.472885 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.472976 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:35:52.473234 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.465778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35779 (local address 127.6.23.193:44249)
0928 14:35:52.466069 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.466083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.466182 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.466414 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.466417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.466621 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:52.466734 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.466742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.467606 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.467610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.468364 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.468372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.468840 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.468843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.468936 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.469489 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:52.469501 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:52.469813 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:52.471585 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.471590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.471592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.471791 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.472075 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.472079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.472081 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.472173 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.472606 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:52.472611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:52.472718 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.472837 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.472996 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:35:52.494294 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35779
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:52.606866 8201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:52.607115 8201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:52.607161 8201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:52.611299 8201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:52.611357 8201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:52.611402 8201 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:35:52.611418 8201 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:35:52.611450 8201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:35:52.616804 8201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:35779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:52.617791 8201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:52.618744 8201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:52.628063 8207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:52.628108 8206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:52.628077 8209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:52.628322 8201 server_base.cc:1034] running on GCE node
I20260928 14:35:52.628787 8201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:52.629474 8201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:52.630661 8201 hybrid_clock.cc:648] HybridClock initialized: now 1790606152630635 us; error 33 us; skew 500 ppm
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:52.633634 8201 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:35:52.634606 8201 webserver.cc:460] Webserver started at http://127.6.23.194:37871/ using document root <none> and password file <none>
I20260928 14:35:52.635172 8201 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:52.635290 8201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:52.635528 8201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:52.637548 8201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "14c33655f9eb4e47af99ded706a370f9"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "09616c7a777c230bd4bcdf0a8c0d840b"
server_key_iv: "36108864c31beaac012d00a30beae0c1"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.638062 8201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "14c33655f9eb4e47af99ded706a370f9"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "09616c7a777c230bd4bcdf0a8c0d840b"
server_key_iv: "36108864c31beaac012d00a30beae0c1"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.641788 8201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:35:52.644145 8216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:52.645661 8201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:35:52.645798 8201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "14c33655f9eb4e47af99ded706a370f9"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "09616c7a777c230bd4bcdf0a8c0d840b"
server_key_iv: "36108864c31beaac012d00a30beae0c1"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.645903 8201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:52.662583 8201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:52.666355 8201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:52.666569 8201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:52.668113 8201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:52.668190 8201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:52.668256 8201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:52.668311 8201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:52.679277 8201 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:45915
I20260928 14:35:52.679536 8331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:45915 every 8 connection(s)
I20260928 14:35:52.680434 8201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:35:52.684154 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.668963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:33069 (local address 127.6.23.254:35779)
0928 14:35:52.669094 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.669098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.670162 (+ 1064us) server_negotiation.cc:407] Connection header received
0928 14:35:52.671261 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.671265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.671333 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.671412 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:52.673255 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.674194 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.676161 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.676379 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.676463 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.679662 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:52.679666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:52.679669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:52.679700 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:52.681507 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.682179 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.682188 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.682190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.682315 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.682714 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.682717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.682719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.683006 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:52.683120 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.683858 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.683986 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:35:52.684962 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.669530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35779 (local address 127.6.23.194:33069)
0928 14:35:52.670005 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.670046 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.670964 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.671890 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.671898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.672401 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:52.673110 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.673133 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.674342 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.674347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.676048 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.676061 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.676558 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.676573 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.676793 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.677587 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:52.677612 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:52.679457 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:52.681670 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.681677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.681694 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.682078 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.682417 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.682421 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.682425 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.682613 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.683135 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:52.683144 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:52.683516 (+ 372us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.683734 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.684075 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260928 14:35:52.686444 8332 heartbeater.cc:347] Connected to a master server at 127.6.23.254:35779
I20260928 14:35:52.686573 8332 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:52.687117 8332 heartbeater.cc:511] Master 127.6.23.254:35779 requested a full tablet report, sending...
I20260928 14:35:52.688263 7989 ts_manager.cc:194] Registered new tserver with Master: 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915)
I20260928 14:35:52.689046 7989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:33069
I20260928 14:35:52.690763 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 8201
I20260928 14:35:52.690900 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:35:52.691186 6239 external_mini_cluster.cc:1428] Setting key 234b46505d560921fe96f520a627ae21
I20260928 14:35:52.696758 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.688878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:50817 (local address 127.6.23.254:35779)
0928 14:35:52.688999 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.689003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.689228 (+ 225us) server_negotiation.cc:407] Connection header received
0928 14:35:52.689349 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.689352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.689390 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.689483 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:52.689959 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.690457 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.691507 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.691778 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.691883 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.693117 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:52.693123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:52.693129 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:52.693162 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:52.694836 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.695384 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.695389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.695393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.695483 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.695870 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.695874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.695876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.696083 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:52.696162 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.696484 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.696587 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:35:52.696889 8338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.688766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35779 (local address 127.6.23.194:50817)
0928 14:35:52.689155 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.689169 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.689267 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.689505 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.689508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.689729 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:52.689840 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.689847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.690617 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.690620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.691393 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.691402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.691890 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.691894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.691974 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.692658 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:52.692670 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:52.692979 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:52.694941 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.694946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.694949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.695277 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.695589 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.695594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.695597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.695729 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.696184 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:52.696190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:52.696313 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.696433 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.696613 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:35:52.719640 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35779
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:52.829056 8341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:52.829319 8341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:52.829406 8341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:52.833444 8341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:52.833513 8341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:52.833554 8341 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:35:52.833607 8341 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:35:52.833683 8341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:35:52.839176 8341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:35779
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:52.840286 8341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:52.841334 8341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:52.851300 8349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:52.851346 8341 server_base.cc:1034] running on GCE node
W20260928 14:35:52.851307 8346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:52.851307 8347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:52.851972 8341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:52.852597 8341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:52.853777 8341 hybrid_clock.cc:648] HybridClock initialized: now 1790606152853745 us; error 45 us; skew 500 ppm
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:52.856606 8341 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:35:52.857486 8341 webserver.cc:460] Webserver started at http://127.6.23.195:34775/ using document root <none> and password file <none>
I20260928 14:35:52.857999 8341 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:52.858055 8341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:52.858214 8341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:52.860234 8341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "45bd86ab9e6640e6bdb348e331880ae3"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "3fbbb806760acc4a784747f13dae21fd"
server_key_iv: "e6e93968cf97045a9e9d75ba5d589422"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.860733 8341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "45bd86ab9e6640e6bdb348e331880ae3"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "3fbbb806760acc4a784747f13dae21fd"
server_key_iv: "e6e93968cf97045a9e9d75ba5d589422"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.864375 8341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:35:52.866798 8356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:52.868095 8341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:52.868218 8341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "45bd86ab9e6640e6bdb348e331880ae3"
format_stamp: "Formatted at 2026-09-28 14:35:52 on dist-test-slave-06lh"
server_key: "3fbbb806760acc4a784747f13dae21fd"
server_key_iv: "e6e93968cf97045a9e9d75ba5d589422"
server_key_version: "encryptionkey@0"
I20260928 14:35:52.868324 8341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:52.891079 8341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:52.894074 8341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:52.894264 8341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:52.895892 8341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:52.895941 8341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:52.896005 8341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:52.896035 8341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:52.906734 8341 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:37471
I20260928 14:35:52.906847 8471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:37471 every 8 connection(s)
I20260928 14:35:52.907837 8341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:35:52.909744 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.896708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:49741 (local address 127.6.23.254:35779)
0928 14:35:52.896853 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.896858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.897694 (+ 836us) server_negotiation.cc:407] Connection header received
0928 14:35:52.898744 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.898749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.898834 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.898897 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:52.900624 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.901286 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.902435 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.902837 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.902964 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.905935 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:52.905939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:52.905941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:52.905971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:52.907668 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.908251 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.908255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.908257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.908365 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.908758 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.908762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.908763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.908982 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:52.909063 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.909530 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.909627 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:35:52.910593 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.897095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35779 (local address 127.6.23.195:49741)
0928 14:35:52.897532 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.897581 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.898451 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.899537 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.899546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.899881 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:52.900485 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.900499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.901411 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.901414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.902296 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.902305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.903041 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.903046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.903283 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.903865 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:52.903880 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:52.905751 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:52.907781 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.907787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.907805 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.908134 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.908486 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.908489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.908491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.908651 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.909090 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:52.909096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:52.909319 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.909481 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.909711 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
I20260928 14:35:52.911813 8472 heartbeater.cc:347] Connected to a master server at 127.6.23.254:35779
I20260928 14:35:52.911965 8472 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:52.912437 8472 heartbeater.cc:511] Master 127.6.23.254:35779 requested a full tablet report, sending...
I20260928 14:35:52.913434 7988 ts_manager.cc:194] Registered new tserver with Master: 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471)
I20260928 14:35:52.914059 7988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:49741
I20260928 14:35:52.916242 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 8341
I20260928 14:35:52.916404 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:35:52.916712 6239 external_mini_cluster.cc:1428] Setting key 1591922c5c20e660526d6ddb17840bd7
I20260928 14:35:52.918740 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:35:52.921542 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.913933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35779 (local address 127.6.23.195:51971)
0928 14:35:52.914277 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.914288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.914398 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.914623 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.914626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.914809 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:52.914905 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.914910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.915905 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.915907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.916858 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.916870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.917410 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.917414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.917529 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.918011 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:52.918022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:52.918306 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:52.920088 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.920093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.920095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.920284 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.920616 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:52.920622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:52.920627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:52.920736 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:52.921159 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:52.921161 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:52.921233 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.921312 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.921417 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":90,"threads_started":1}
I20260928 14:35:52.921698 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.914010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:51971 (local address 127.6.23.254:35779)
0928 14:35:52.914142 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.914146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.914358 (+ 212us) server_negotiation.cc:407] Connection header received
0928 14:35:52.914490 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.914492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.914525 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.914582 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:52.915022 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.915756 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.916967 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.917297 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.917406 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.918447 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:52.918452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:52.918455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:52.918481 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:52.919986 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.920381 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.920386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.920387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.920490 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.920831 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.920836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.920837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.921068 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:52.921170 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.921398 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.921518 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:35:52 dist-test-slave-06lh krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:52.944698 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.936776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.6.23.254:35779)
0928 14:35:52.936905 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.936908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.936920 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:35:52.936959 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.936962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.936997 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.937091 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:52.937806 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.938331 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.939172 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.939459 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.939569 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.942122 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:52.942125 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:52.942127 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:52.942149 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:52.943385 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.943749 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.943751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.943752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.943821 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:52.944044 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:52.944046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:52.944047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:52.944206 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:52.944297 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.944491 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.944578 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:35:52.947745 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:35:52.950845 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:52.965304 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.959443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37471 (local address 127.0.0.1:45678)
0928 14:35:52.960228 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.960241 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.960308 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.961291 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.961293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.961311 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:52.961404 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.961410 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.962626 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.962632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.963902 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.963912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.964765 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.964768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.964880 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.964921 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.965041 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.965159 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":6,"thread_start_us":84,"threads_started":1}
I20260928 14:35:52.965313 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.959835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.6.23.195:37471)
0928 14:35:52.960866 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.960878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.960899 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:35:52.960971 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.960977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.961139 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.961290 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:52.961559 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.962519 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.964021 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.964643 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.964766 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.964930 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.965075 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.965162 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":99,"threads_started":1}
I20260928 14:35:52.965559 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.959176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.6.23.194:45915)
0928 14:35:52.959810 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.959823 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.960100 (+ 277us) server_negotiation.cc:407] Connection header received
0928 14:35:52.960182 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.960187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.960405 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.960575 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:52.960866 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.962108 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.963897 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.964666 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.964795 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.964945 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.965308 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.965408 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":162,"threads_started":1}
I20260928 14:35:52.965579 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.959198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41103 (local address 127.0.0.1:56376)
0928 14:35:52.960027 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.960040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.960131 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.960830 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.960833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.960853 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:52.960969 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.960974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.962163 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.962169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.964163 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.964172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.965235 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.965239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.965336 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.965356 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.965400 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.965458 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":127,"thread_start_us":109,"threads_started":1}
I20260928 14:35:52.965953 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.959007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:45915 (local address 127.0.0.1:34428)
0928 14:35:52.959749 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:52.959802 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:52.959953 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:52.960553 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:52.960559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:52.960583 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:52.960733 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.960740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.962204 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.962207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.963770 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:52.963780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.965044 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.965047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:52.965142 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.965179 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:35:52.965758 (+ 579us) client_negotiation.cc:240] Negotiation successful
0928 14:35:52.965831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":116,"thread_start_us":88,"threads_started":1}
I20260928 14:35:52.966116 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:52.959878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56376 (local address 127.6.23.193:41103)
0928 14:35:52.960374 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:52.960389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:52.960411 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:35:52.960496 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:52.960503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:52.960682 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:52.961084 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:52.961181 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.962051 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.964285 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:52.965105 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:52.965677 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:52.965823 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:52.965872 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 14:35:52.965955 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":91,"threads_started":1}
I20260928 14:35:52.968636 8126 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce51e9f2650475b825aa817a2c818b2 (DEFAULT_TABLE table=test-table [id=f8b8732086ab46d7ade9bbe2bfce6100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:52.968830 8266 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce51e9f2650475b825aa817a2c818b2 (DEFAULT_TABLE table=test-table [id=f8b8732086ab46d7ade9bbe2bfce6100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:52.968746 8406 tablet_service.cc:1467] Processing CreateTablet for tablet 6ce51e9f2650475b825aa817a2c818b2 (DEFAULT_TABLE table=test-table [id=f8b8732086ab46d7ade9bbe2bfce6100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:52.969626 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce51e9f2650475b825aa817a2c818b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:52.969709 8266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce51e9f2650475b825aa817a2c818b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:52.969722 8406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ce51e9f2650475b825aa817a2c818b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:52.977139 8496 tablet_bootstrap.cc:492] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Bootstrap starting.
I20260928 14:35:52.977160 8497 tablet_bootstrap.cc:492] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Bootstrap starting.
I20260928 14:35:52.977922 8495 tablet_bootstrap.cc:492] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Bootstrap starting.
I20260928 14:35:52.979493 8496 tablet_bootstrap.cc:654] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:52.979493 8497 tablet_bootstrap.cc:654] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:52.979614 8495 tablet_bootstrap.cc:654] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:52.980387 8496 log.cc:824] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:52.980387 8497 log.cc:824] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:52.980784 8495 log.cc:824] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:52.982424 8497 tablet_bootstrap.cc:492] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: No bootstrap required, opened a new log
I20260928 14:35:52.982453 8496 tablet_bootstrap.cc:492] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: No bootstrap required, opened a new log
I20260928 14:35:52.982630 8497 ts_tablet_manager.cc:1397] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260928 14:35:52.982637 8496 ts_tablet_manager.cc:1397] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 14:35:52.984699 8495 tablet_bootstrap.cc:492] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: No bootstrap required, opened a new log
I20260928 14:35:52.984908 8495 ts_tablet_manager.cc:1397] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:35:52.986270 8496 raft_consensus.cc:348] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:52.986511 8496 raft_consensus.cc:374] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:52.986395 8497 raft_consensus.cc:348] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:52.986573 8496 raft_consensus.cc:729] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c33655f9eb4e47af99ded706a370f9, State: Initialized, Role: FOLLOWER
I20260928 14:35:52.986629 8497 raft_consensus.cc:374] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:52.986697 8497 raft_consensus.cc:729] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bd86ab9e6640e6bdb348e331880ae3, State: Initialized, Role: FOLLOWER
I20260928 14:35:52.987062 8496 consensus_queue.cc:260] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:52.987161 8497 consensus_queue.cc:260] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:52.987778 8332 heartbeater.cc:503] Master 127.6.23.254:35779 was elected leader, sending a full tablet report...
I20260928 14:35:52.988140 8496 ts_tablet_manager.cc:1428] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:35:52.988140 8497 ts_tablet_manager.cc:1428] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260928 14:35:52.988449 8472 heartbeater.cc:503] Master 127.6.23.254:35779 was elected leader, sending a full tablet report...
I20260928 14:35:52.988696 8495 raft_consensus.cc:348] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:52.988929 8495 raft_consensus.cc:374] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:52.988976 8495 raft_consensus.cc:729] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a762fc6629d414e8e61f52f6e39f5b3, State: Initialized, Role: FOLLOWER
I20260928 14:35:52.989389 8495 consensus_queue.cc:260] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:52.990272 8495 ts_tablet_manager.cc:1428] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:35:52.990317 8192 heartbeater.cc:503] Master 127.6.23.254:35779 was elected leader, sending a full tablet report...
I20260928 14:35:53.038913 8503 raft_consensus.cc:482] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:53.039130 8503 raft_consensus.cc:504] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.040333 8503 leader_election.cc:288] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915), 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471)
I20260928 14:35:53.046005 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.040702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:38339 (local address 127.6.23.194:45915)
0928 14:35:53.040857 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.040862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.041121 (+ 259us) server_negotiation.cc:407] Connection header received
0928 14:35:53.041192 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.041195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.041252 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.041346 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:53.041731 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.042615 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.044540 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.045234 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.045360 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.045419 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.045751 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.045830 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:35:53.046005 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.040746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:53451 (local address 127.6.23.195:37471)
0928 14:35:53.040942 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.040953 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.041121 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:35:53.041192 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.041195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.041252 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.041346 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:53.041731 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.042636 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.044541 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.045236 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.045384 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.045431 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.045751 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.045830 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 14:35:53.046042 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.040560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:45915 (local address 127.6.23.193:38339)
0928 14:35:53.040965 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.040980 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.041107 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.041367 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.041371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.041409 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:53.041606 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.041613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.042740 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.042743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.044399 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.044411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.045370 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.045374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.045584 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.045628 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.045708 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.045833 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":243,"spinlock_wait_cycles":6528,"thread_start_us":154,"threads_started":1}
I20260928 14:35:53.046047 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.040602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37471 (local address 127.6.23.193:53451)
0928 14:35:53.040999 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.041028 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.041114 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.041368 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.041371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.041408 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:53.041607 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.041613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.042740 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.042743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.044399 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.044411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.045370 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.045374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.045579 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.045632 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.045708 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.045834 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":352,"spinlock_wait_cycles":69248,"thread_start_us":122,"threads_started":1}
I20260928 14:35:53.047108 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce51e9f2650475b825aa817a2c818b2" candidate_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bd86ab9e6640e6bdb348e331880ae3" is_pre_election: true
I20260928 14:35:53.047147 8286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce51e9f2650475b825aa817a2c818b2" candidate_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c33655f9eb4e47af99ded706a370f9" is_pre_election: true
I20260928 14:35:53.047487 8425 raft_consensus.cc:2454] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a762fc6629d414e8e61f52f6e39f5b3 in term 0.
I20260928 14:35:53.047488 8286 raft_consensus.cc:2454] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a762fc6629d414e8e61f52f6e39f5b3 in term 0.
I20260928 14:35:53.048003 8077 leader_election.cc:302] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14c33655f9eb4e47af99ded706a370f9, 3a762fc6629d414e8e61f52f6e39f5b3, 45bd86ab9e6640e6bdb348e331880ae3; no voters:
I20260928 14:35:53.048305 8503 raft_consensus.cc:2781] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:53.048396 8503 raft_consensus.cc:482] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:53.048445 8503 raft_consensus.cc:3037] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.049553 8503 raft_consensus.cc:504] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.049989 8503 leader_election.cc:288] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [CANDIDATE]: Term 1 election: Requested vote from peers 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915), 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471)
I20260928 14:35:53.050335 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce51e9f2650475b825aa817a2c818b2" candidate_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bd86ab9e6640e6bdb348e331880ae3"
I20260928 14:35:53.050335 8286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ce51e9f2650475b825aa817a2c818b2" candidate_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c33655f9eb4e47af99ded706a370f9"
I20260928 14:35:53.050472 8425 raft_consensus.cc:3037] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.050475 8286 raft_consensus.cc:3037] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.052274 8286 raft_consensus.cc:2454] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a762fc6629d414e8e61f52f6e39f5b3 in term 1.
I20260928 14:35:53.052266 8425 raft_consensus.cc:2454] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a762fc6629d414e8e61f52f6e39f5b3 in term 1.
I20260928 14:35:53.052608 8078 leader_election.cc:302] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a762fc6629d414e8e61f52f6e39f5b3, 45bd86ab9e6640e6bdb348e331880ae3; no voters:
I20260928 14:35:53.052803 8503 raft_consensus.cc:2781] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:53.053073 8503 raft_consensus.cc:686] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3a762fc6629d414e8e61f52f6e39f5b3, State: Running, Role: LEADER
I20260928 14:35:53.053445 8503 consensus_queue.cc:237] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.056860 7988 catalog_manager.cc:5495] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:35:53 dist-test-slave-06lh krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606152, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:53.109472 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.100945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35779 (local address 127.0.0.1:57814)
0928 14:35:53.101260 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.101272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.101407 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.101694 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.101697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.102135 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:53.102224 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.102230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.102959 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.102964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.103602 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.103611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.104034 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.104037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.104132 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.104678 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:53.104690 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:53.106155 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:53.107859 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:53.107870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:53.107877 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:53.108107 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:53.108382 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:53.108385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:53.108387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:53.108464 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:53.108863 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:53.108871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:53.108986 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.109107 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.109266 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260928 14:35:53.109469 8038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.100980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.6.23.254:35779)
0928 14:35:53.101132 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.101137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.101350 (+ 213us) server_negotiation.cc:407] Connection header received
0928 14:35:53.101537 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.101542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.101588 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.101674 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:53.102338 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.102854 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.103716 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.103924 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.104038 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.106289 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:53.106294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:53.106301 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:53.106322 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:53.107753 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:53.108207 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:53.108213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:53.108215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:53.108288 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:53.108577 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:53.108583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:53.108586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:53.108756 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:53.108863 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.109174 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.109315 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":52}
I20260928 14:35:53.113449 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:57814:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:35:53.114113 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:53.119042 8266 tablet_service.cc:1467] Processing CreateTablet for tablet f794539044b74bdabf5fc88efa5e7ab4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ccd69dab3034428db3226bce44bf7670]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:35:53.119082 8406 tablet_service.cc:1467] Processing CreateTablet for tablet f794539044b74bdabf5fc88efa5e7ab4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ccd69dab3034428db3226bce44bf7670]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:35:53.118989 8126 tablet_service.cc:1467] Processing CreateTablet for tablet f794539044b74bdabf5fc88efa5e7ab4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ccd69dab3034428db3226bce44bf7670]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:35:53.119750 8266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f794539044b74bdabf5fc88efa5e7ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:53.119748 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f794539044b74bdabf5fc88efa5e7ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:53.119748 8406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f794539044b74bdabf5fc88efa5e7ab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:53.122171 8496 tablet_bootstrap.cc:492] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: Bootstrap starting.
I20260928 14:35:53.122648 8497 tablet_bootstrap.cc:492] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: Bootstrap starting.
I20260928 14:35:53.123008 8495 tablet_bootstrap.cc:492] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: Bootstrap starting.
I20260928 14:35:53.123194 8496 tablet_bootstrap.cc:654] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:53.123663 8497 tablet_bootstrap.cc:654] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:53.124212 8495 tablet_bootstrap.cc:654] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:53.124598 8496 tablet_bootstrap.cc:492] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: No bootstrap required, opened a new log
I20260928 14:35:53.124692 8496 ts_tablet_manager.cc:1397] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:35:53.124893 8497 tablet_bootstrap.cc:492] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: No bootstrap required, opened a new log
I20260928 14:35:53.124979 8497 ts_tablet_manager.cc:1397] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:35:53.125175 8496 raft_consensus.cc:348] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.125293 8496 raft_consensus.cc:374] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:53.125329 8496 raft_consensus.cc:729] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c33655f9eb4e47af99ded706a370f9, State: Initialized, Role: FOLLOWER
I20260928 14:35:53.125427 8497 raft_consensus.cc:348] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.125464 8496 consensus_queue.cc:260] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.125532 8497 raft_consensus.cc:374] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:53.125567 8497 raft_consensus.cc:729] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bd86ab9e6640e6bdb348e331880ae3, State: Initialized, Role: FOLLOWER
I20260928 14:35:53.125692 8497 consensus_queue.cc:260] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.125747 8501 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.125873 8501 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:53.125927 8495 tablet_bootstrap.cc:492] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: No bootstrap required, opened a new log
I20260928 14:35:53.125999 8495 ts_tablet_manager.cc:1397] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:35:53.125974 8502 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.126106 8502 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:53.126099 8496 ts_tablet_manager.cc:1428] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 14:35:53.126471 8495 raft_consensus.cc:348] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.126585 8495 raft_consensus.cc:374] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:53.126623 8495 raft_consensus.cc:729] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a762fc6629d414e8e61f52f6e39f5b3, State: Initialized, Role: FOLLOWER
I20260928 14:35:53.127023 8495 consensus_queue.cc:260] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.127381 8495 ts_tablet_manager.cc:1428] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:53.127311 8503 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.127465 8503 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:53.127581 8497 ts_tablet_manager.cc:1428] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
W20260928 14:35:53.158828 8473 tablet.cc:2369] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:35:53.161011 8502 raft_consensus.cc:482] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:53.161119 8502 raft_consensus.cc:504] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.162230 8502 leader_election.cc:288] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103), 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915)
I20260928 14:35:53.166266 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.162449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:45915 (local address 127.6.23.195:47899)
0928 14:35:53.162666 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.162682 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.162788 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.163076 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.163079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.163098 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:53.163261 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.163272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.164126 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.164130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.165113 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.165122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.165842 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.165845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.165940 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.165966 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.166058 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.166128 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260928 14:35:53.166355 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.162543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:47899 (local address 127.6.23.194:45915)
0928 14:35:53.162665 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.162669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.162760 (+ 91us) server_negotiation.cc:407] Connection header received
0928 14:35:53.162904 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.162907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.162973 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.163053 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:53.163399 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.163994 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.165246 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.165727 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.165817 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.165861 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.166143 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.166225 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:35:53.166646 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.162543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:52479 (local address 127.6.23.193:41103)
0928 14:35:53.162664 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.162669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.163045 (+ 376us) server_negotiation.cc:407] Connection header received
0928 14:35:53.163111 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.163114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.163164 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.163286 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:53.163594 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.164211 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.165457 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.165947 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.166108 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.166152 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.166451 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.166534 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:35:53.166924 8286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f794539044b74bdabf5fc88efa5e7ab4" candidate_uuid: "45bd86ab9e6640e6bdb348e331880ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c33655f9eb4e47af99ded706a370f9" is_pre_election: true
I20260928 14:35:53.167037 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.162449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41103 (local address 127.6.23.195:52479)
0928 14:35:53.162909 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.162923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.163037 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.163310 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.163316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.163339 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:53.163469 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.163476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.164331 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.164334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.165317 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.165327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.166170 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.166174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.166268 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.166291 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.166806 (+ 515us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.166883 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":33,"thread_start_us":118,"threads_started":1}
I20260928 14:35:53.167104 8286 raft_consensus.cc:2454] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45bd86ab9e6640e6bdb348e331880ae3 in term 0.
I20260928 14:35:53.167542 8357 leader_election.cc:302] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c33655f9eb4e47af99ded706a370f9, 45bd86ab9e6640e6bdb348e331880ae3; no voters:
I20260928 14:35:53.167630 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f794539044b74bdabf5fc88efa5e7ab4" candidate_uuid: "45bd86ab9e6640e6bdb348e331880ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" is_pre_election: true
I20260928 14:35:53.167850 8502 raft_consensus.cc:2781] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:53.167919 8502 raft_consensus.cc:482] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:53.167893 8146 raft_consensus.cc:2454] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45bd86ab9e6640e6bdb348e331880ae3 in term 0.
I20260928 14:35:53.167970 8502 raft_consensus.cc:3037] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.169054 8502 raft_consensus.cc:504] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.169395 8502 leader_election.cc:288] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [CANDIDATE]: Term 1 election: Requested vote from peers 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103), 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915)
I20260928 14:35:53.169698 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f794539044b74bdabf5fc88efa5e7ab4" candidate_uuid: "45bd86ab9e6640e6bdb348e331880ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a762fc6629d414e8e61f52f6e39f5b3"
I20260928 14:35:53.169786 8286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f794539044b74bdabf5fc88efa5e7ab4" candidate_uuid: "45bd86ab9e6640e6bdb348e331880ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c33655f9eb4e47af99ded706a370f9"
I20260928 14:35:53.169826 8146 raft_consensus.cc:3037] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.169924 8286 raft_consensus.cc:3037] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.170885 8146 raft_consensus.cc:2454] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bd86ab9e6640e6bdb348e331880ae3 in term 1.
I20260928 14:35:53.171026 8286 raft_consensus.cc:2454] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bd86ab9e6640e6bdb348e331880ae3 in term 1.
I20260928 14:35:53.171218 8359 leader_election.cc:302] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a762fc6629d414e8e61f52f6e39f5b3, 45bd86ab9e6640e6bdb348e331880ae3; no voters:
I20260928 14:35:53.171429 8502 raft_consensus.cc:2781] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:53.171653 8502 raft_consensus.cc:686] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 LEADER]: Becoming Leader. State: Replica: 45bd86ab9e6640e6bdb348e331880ae3, State: Running, Role: LEADER
I20260928 14:35:53.171993 8502 consensus_queue.cc:237] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.174186 8518 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: TxnStatusTablet state changed. Reason: New leader 45bd86ab9e6640e6bdb348e331880ae3. Latest consensus state: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.174371 8518 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:35:53.174685 8520 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:35:53.174919 8520 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:35:53.179711 7989 catalog_manager.cc:5495] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195). New cstate: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } health_report { overall_health: HEALTHY } } }
W20260928 14:35:53.181777 8333 tablet.cc:2369] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:35:53.198025 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.193760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37471 (local address 127.0.0.1:45688)
0928 14:35:53.193893 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.193910 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.194019 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.194381 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.194383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.194390 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:53.194479 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.194483 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.195511 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.195514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.196319 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.196329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.196885 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.196888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.196957 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.197001 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.197763 (+ 762us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.197778 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.197834 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.197902 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 14:35:53.198058 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.193863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.6.23.195:37471)
0928 14:35:53.194001 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.194005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.194042 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:35:53.194196 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.194199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.194263 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.194368 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:53.194578 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.195407 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.196426 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.196763 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.196862 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.197143 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.197667 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.197778 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.197849 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.197911 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:35:53.203035 8286 raft_consensus.cc:1261] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Refusing update from remote peer 45bd86ab9e6640e6bdb348e331880ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:53.203071 8146 raft_consensus.cc:1261] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Refusing update from remote peer 45bd86ab9e6640e6bdb348e331880ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:53.203748 8502 consensus_queue.cc:1035] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:53.203915 8518 consensus_queue.cc:1035] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:53.207021 8503 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: TxnStatusTablet state changed. Reason: New leader 45bd86ab9e6640e6bdb348e331880ae3. Latest consensus state: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.207132 8503 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:53.207134 8501 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: TxnStatusTablet state changed. Reason: New leader 45bd86ab9e6640e6bdb348e331880ae3. Latest consensus state: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.207269 8501 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:53.210260 8518 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.210402 8518 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:35:53.210585 8523 mvcc.cc:205] Tried to move back new op lower bound from 7334322803512766464 to 7334322803395170304. Current Snapshot: MvccSnapshot[applied={T|T < 7334322803512766464}]
W20260928 14:35:53.210657 8193 tablet.cc:2369] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:35:53.211169 8501 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.211298 8501 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 14c33655f9eb4e47af99ded706a370f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:53.212093 8524 mvcc.cc:205] Tried to move back new op lower bound from 7334322803512766464 to 7334322803395170304. Current Snapshot: MvccSnapshot[applied={T|T < 7334322803512766464}]
I20260928 14:35:53.216648 8518 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.216758 8518 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 45bd86ab9e6640e6bdb348e331880ae3: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:35:53.218036 8503 tablet_replica.cc:439] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } } }
I20260928 14:35:53.218146 8503 tablet_replica.cc:442] T f794539044b74bdabf5fc88efa5e7ab4 P 3a762fc6629d414e8e61f52f6e39f5b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:35:53.223410 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.219614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.6.23.193:41103)
0928 14:35:53.219715 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.219720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.219734 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:35:53.219805 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.219808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.219868 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.219981 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:53.220246 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.221038 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.221864 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.222214 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.222315 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.222543 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.223041 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.223149 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.223187 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.223277 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:35:53.235404 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.231470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:56337 (local address 127.6.23.195:37471)
0928 14:35:53.231611 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.231615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.231829 (+ 214us) server_negotiation.cc:407] Connection header received
0928 14:35:53.231974 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.231976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.232030 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.232088 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:53.232356 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.232837 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.234146 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.234357 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.234489 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.234871 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.234959 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.235046 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.235201 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.235287 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:35:53.235842 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.231380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37471 (local address 127.6.23.193:56337)
0928 14:35:53.231746 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.231761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.231862 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.232121 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.232124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.232132 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:53.232260 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.232266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.232940 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.232944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.234000 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.234012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.234491 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.234495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.234610 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.234685 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.235048 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.235057 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.235624 (+ 567us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.235709 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":107,"threads_started":1}
I20260928 14:35:53.242672 8286 raft_consensus.cc:1261] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Refusing update from remote peer 3a762fc6629d414e8e61f52f6e39f5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:53.242846 8425 raft_consensus.cc:1261] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Refusing update from remote peer 3a762fc6629d414e8e61f52f6e39f5b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:53.243129 8506 consensus_queue.cc:1035] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:53.243350 8503 consensus_queue.cc:1035] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:53.247080 8525 mvcc.cc:205] Tried to move back new op lower bound from 7334322803677331456 to 7334322802910146560. Current Snapshot: MvccSnapshot[applied={T|T < 7334322803677331456}]
I20260928 14:35:53.267282 8494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.263692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:43129 (local address 127.6.23.193:41103)
0928 14:35:53.263818 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.263824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.263838 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:35:53.263881 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.263884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.263942 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.264013 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:53.264347 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.264854 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.266045 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.266393 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.266492 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.266782 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.266866 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.266956 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.267052 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.267111 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:35:53.267283 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.263463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41103 (local address 127.6.23.195:43129)
0928 14:35:53.263623 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.263641 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.263731 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.264134 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.264137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.264145 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:53.264254 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.264260 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.264936 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.264940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.265910 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.265920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.266499 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.266506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.266607 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.266658 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.266962 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.266969 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.267039 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.267117 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 14:35:53.295424 7987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:53.295639 7987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:53.298491 7987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: Sending DeleteTablet for 3 replicas of tablet 6ce51e9f2650475b825aa817a2c818b2
I20260928 14:35:53.299306 8266 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce51e9f2650475b825aa817a2c818b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:53 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34428
I20260928 14:35:53.299360 8406 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce51e9f2650475b825aa817a2c818b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:53 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:45678
I20260928 14:35:53.300019 8543 tablet_replica.cc:330] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: stopping tablet replica
I20260928 14:35:53.300279 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ce51e9f2650475b825aa817a2c818b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:53 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:56376
I20260928 14:35:53.300757 8542 tablet_replica.cc:330] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: stopping tablet replica
I20260928 14:35:53.301076 8542 raft_consensus.cc:2229] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:53.300993 7987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:35:53.301288 8542 raft_consensus.cc:2258] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:35:53.301394 7987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:53.301643 8544 tablet_replica.cc:330] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: stopping tablet replica
I20260928 14:35:53.301910 8544 raft_consensus.cc:2229] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:35:53.302249 8544 raft_consensus.cc:2258] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:53.302969 8543 raft_consensus.cc:2229] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:53.303185 8543 raft_consensus.cc:2258] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:53.304791 8543 ts_tablet_manager.cc:1905] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:53.305967 8542 ts_tablet_manager.cc:1905] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:53.306977 8266 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd982f07a24450da8ab156f8772ba99 (DEFAULT_TABLE table=test-table [id=25477b05c9f64141874498cbf6960267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:53.307272 8266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd982f07a24450da8ab156f8772ba99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:53.308161 8544 ts_tablet_manager.cc:1905] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:53.308341 8406 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd982f07a24450da8ab156f8772ba99 (DEFAULT_TABLE table=test-table [id=25477b05c9f64141874498cbf6960267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:53.308599 8406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd982f07a24450da8ab156f8772ba99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:53.308581 8543 ts_tablet_manager.cc:1918] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:53.308768 8543 log.cc:1196] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/6ce51e9f2650475b825aa817a2c818b2
I20260928 14:35:53.309176 8543 ts_tablet_manager.cc:1939] T 6ce51e9f2650475b825aa817a2c818b2 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting consensus metadata
I20260928 14:35:53.310731 7974 catalog_manager.cc:4841] TS 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471): tablet 6ce51e9f2650475b825aa817a2c818b2 (table test-table [id=f8b8732086ab46d7ade9bbe2bfce6100]) successfully deleted
I20260928 14:35:53.310871 8497 tablet_bootstrap.cc:492] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: Bootstrap starting.
I20260928 14:35:53.311172 8544 ts_tablet_manager.cc:1918] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:53.311311 8544 log.cc:1196] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/6ce51e9f2650475b825aa817a2c818b2
I20260928 14:35:53.311899 8497 tablet_bootstrap.cc:654] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:53.313161 8126 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd982f07a24450da8ab156f8772ba99 (DEFAULT_TABLE table=test-table [id=25477b05c9f64141874498cbf6960267]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:53.313413 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd982f07a24450da8ab156f8772ba99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:53.313726 8542 ts_tablet_manager.cc:1918] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:53.313828 8542 log.cc:1196] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/6ce51e9f2650475b825aa817a2c818b2
I20260928 14:35:53.314141 8542 ts_tablet_manager.cc:1939] T 6ce51e9f2650475b825aa817a2c818b2 P 14c33655f9eb4e47af99ded706a370f9: Deleting consensus metadata
I20260928 14:35:53.315436 7973 catalog_manager.cc:4841] TS 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915): tablet 6ce51e9f2650475b825aa817a2c818b2 (table test-table [id=f8b8732086ab46d7ade9bbe2bfce6100]) successfully deleted
I20260928 14:35:53.315755 8495 tablet_bootstrap.cc:492] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: Bootstrap starting.
I20260928 14:35:53.315989 8544 ts_tablet_manager.cc:1939] T 6ce51e9f2650475b825aa817a2c818b2 P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting consensus metadata
I20260928 14:35:53.316706 8495 tablet_bootstrap.cc:654] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:53.317759 8496 tablet_bootstrap.cc:492] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: Bootstrap starting.
I20260928 14:35:53.318145 8495 tablet_bootstrap.cc:492] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: No bootstrap required, opened a new log
I20260928 14:35:53.318216 8495 ts_tablet_manager.cc:1397] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:35:53.318661 8496 tablet_bootstrap.cc:654] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:53.318630 8495 raft_consensus.cc:348] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.318770 8495 raft_consensus.cc:374] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:53.318802 8495 raft_consensus.cc:729] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a762fc6629d414e8e61f52f6e39f5b3, State: Initialized, Role: FOLLOWER
I20260928 14:35:53.318926 8495 consensus_queue.cc:260] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.319170 8495 ts_tablet_manager.cc:1428] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:53.319674 7975 catalog_manager.cc:4841] TS 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103): tablet 6ce51e9f2650475b825aa817a2c818b2 (table test-table [id=f8b8732086ab46d7ade9bbe2bfce6100]) successfully deleted
I20260928 14:35:53.321378 8496 tablet_bootstrap.cc:492] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: No bootstrap required, opened a new log
I20260928 14:35:53.321470 8496 ts_tablet_manager.cc:1397] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 14:35:53.321915 8496 raft_consensus.cc:348] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.322024 8496 raft_consensus.cc:374] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:53.322055 8496 raft_consensus.cc:729] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c33655f9eb4e47af99ded706a370f9, State: Initialized, Role: FOLLOWER
I20260928 14:35:53.322172 8496 consensus_queue.cc:260] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.322422 8496 ts_tablet_manager.cc:1428] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:35:53.322574 8497 tablet_bootstrap.cc:492] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: No bootstrap required, opened a new log
I20260928 14:35:53.322669 8497 ts_tablet_manager.cc:1397] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260928 14:35:53.323091 8497 raft_consensus.cc:348] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.323192 8497 raft_consensus.cc:374] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:53.323221 8497 raft_consensus.cc:729] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bd86ab9e6640e6bdb348e331880ae3, State: Initialized, Role: FOLLOWER
I20260928 14:35:53.323380 8497 consensus_queue.cc:260] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.323630 8497 ts_tablet_manager.cc:1428] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:53.477419 8502 raft_consensus.cc:482] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:53.477573 8502 raft_consensus.cc:504] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.478019 8502 leader_election.cc:288] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103), 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915)
I20260928 14:35:53.478485 8286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd982f07a24450da8ab156f8772ba99" candidate_uuid: "45bd86ab9e6640e6bdb348e331880ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c33655f9eb4e47af99ded706a370f9" is_pre_election: true
I20260928 14:35:53.478487 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd982f07a24450da8ab156f8772ba99" candidate_uuid: "45bd86ab9e6640e6bdb348e331880ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" is_pre_election: true
I20260928 14:35:53.478642 8286 raft_consensus.cc:2454] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45bd86ab9e6640e6bdb348e331880ae3 in term 0.
I20260928 14:35:53.478658 8146 raft_consensus.cc:2454] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45bd86ab9e6640e6bdb348e331880ae3 in term 0.
I20260928 14:35:53.478982 8359 leader_election.cc:302] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a762fc6629d414e8e61f52f6e39f5b3, 45bd86ab9e6640e6bdb348e331880ae3; no voters:
I20260928 14:35:53.479156 8502 raft_consensus.cc:2781] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:53.479213 8502 raft_consensus.cc:482] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:53.479314 8502 raft_consensus.cc:3037] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.480223 8502 raft_consensus.cc:504] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.480557 8502 leader_election.cc:288] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [CANDIDATE]: Term 1 election: Requested vote from peers 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103), 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915)
I20260928 14:35:53.480867 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd982f07a24450da8ab156f8772ba99" candidate_uuid: "45bd86ab9e6640e6bdb348e331880ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a762fc6629d414e8e61f52f6e39f5b3"
I20260928 14:35:53.480918 8286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd982f07a24450da8ab156f8772ba99" candidate_uuid: "45bd86ab9e6640e6bdb348e331880ae3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c33655f9eb4e47af99ded706a370f9"
I20260928 14:35:53.480993 8146 raft_consensus.cc:3037] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.481043 8286 raft_consensus.cc:3037] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:53.481876 8286 raft_consensus.cc:2454] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bd86ab9e6640e6bdb348e331880ae3 in term 1.
I20260928 14:35:53.481980 8146 raft_consensus.cc:2454] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45bd86ab9e6640e6bdb348e331880ae3 in term 1.
I20260928 14:35:53.482197 8357 leader_election.cc:302] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c33655f9eb4e47af99ded706a370f9, 45bd86ab9e6640e6bdb348e331880ae3; no voters:
I20260928 14:35:53.482425 8502 raft_consensus.cc:2781] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:53.482568 8502 raft_consensus.cc:686] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 LEADER]: Becoming Leader. State: Replica: 45bd86ab9e6640e6bdb348e331880ae3, State: Running, Role: LEADER
I20260928 14:35:53.482710 8502 consensus_queue.cc:237] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:53.484386 7988 catalog_manager.cc:5495] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195). New cstate: current_term: 1 leader_uuid: "45bd86ab9e6640e6bdb348e331880ae3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } health_report { overall_health: HEALTHY } } }
I20260928 14:35:53.515987 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.512785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.6.23.195:37471)
0928 14:35:53.512904 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.512909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.513024 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:35:53.513180 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.513183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.513235 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.513314 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:53.513579 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.514080 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.514910 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.515144 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.515279 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.515537 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.515628 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.515699 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.515794 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.515844 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:35:53.524312 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.521682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:56417 (local address 127.6.23.195:37471)
0928 14:35:53.521797 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:53.521800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:53.521812 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:35:53.521901 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:53.521903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:53.521940 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:53.522001 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:53.522184 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.522649 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.523455 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.523661 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.523740 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.523939 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.523999 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.524052 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:53.524128 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 14:35:53.524181 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":48}
I20260928 14:35:53.524312 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:53.521623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37471 (local address 127.6.23.195:56417)
0928 14:35:53.521762 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:53.521774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:53.521838 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:53.522026 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:53.522028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:53.522033 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:53.522121 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.522125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.522726 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.522728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.523348 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:53.523357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:53.523759 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:53.523762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:53.523816 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:53.523838 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:53.524069 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:53.524073 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:35:53.524121 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 14:35:53.524201 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":17}
I20260928 14:35:53.529874 8146 raft_consensus.cc:1261] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Refusing update from remote peer 45bd86ab9e6640e6bdb348e331880ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:53.529964 8286 raft_consensus.cc:1261] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Refusing update from remote peer 45bd86ab9e6640e6bdb348e331880ae3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:53.530356 8535 consensus_queue.cc:1035] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:53.530548 8502 consensus_queue.cc:1035] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:35:53.544533 8523 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:35:55.099181 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:55.099352 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:55.101747 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: Sending DeleteTablet for 3 replicas of tablet 8cd982f07a24450da8ab156f8772ba99
I20260928 14:35:55.102346 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd982f07a24450da8ab156f8772ba99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:45678
I20260928 14:35:55.102329 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd982f07a24450da8ab156f8772ba99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:56376
I20260928 14:35:55.102367 8266 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cd982f07a24450da8ab156f8772ba99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34428
I20260928 14:35:55.102699 8575 tablet_replica.cc:330] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: stopping tablet replica
I20260928 14:35:55.102816 8574 tablet_replica.cc:330] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: stopping tablet replica
I20260928 14:35:55.102975 8574 raft_consensus.cc:2229] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:55.103008 8575 raft_consensus.cc:2229] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:55.103118 8574 raft_consensus.cc:2258] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:55.103144 8575 raft_consensus.cc:2258] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:55.103869 8575 ts_tablet_manager.cc:1905] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:55.104596 8573 tablet_replica.cc:330] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: stopping tablet replica
I20260928 14:35:55.104725 8573 raft_consensus.cc:2229] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:35:55.105043 8573 raft_consensus.cc:2258] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:55.105644 8574 ts_tablet_manager.cc:1905] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:55.106351 8573 ts_tablet_manager.cc:1905] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:55.106804 8575 ts_tablet_manager.cc:1918] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:35:55.106894 8575 log.cc:1196] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/8cd982f07a24450da8ab156f8772ba99
I20260928 14:35:55.107175 8575 ts_tablet_manager.cc:1939] T 8cd982f07a24450da8ab156f8772ba99 P 14c33655f9eb4e47af99ded706a370f9: Deleting consensus metadata
I20260928 14:35:55.107931 7973 catalog_manager.cc:4841] TS 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915): tablet 8cd982f07a24450da8ab156f8772ba99 (table test-table [id=25477b05c9f64141874498cbf6960267]) successfully deleted
I20260928 14:35:55.108349 8574 ts_tablet_manager.cc:1918] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:35:55.108428 8574 log.cc:1196] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/8cd982f07a24450da8ab156f8772ba99
I20260928 14:35:55.108683 8574 ts_tablet_manager.cc:1939] T 8cd982f07a24450da8ab156f8772ba99 P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting consensus metadata
I20260928 14:35:55.109380 7975 catalog_manager.cc:4841] TS 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103): tablet 8cd982f07a24450da8ab156f8772ba99 (table test-table [id=25477b05c9f64141874498cbf6960267]) successfully deleted
I20260928 14:35:55.109583 8573 ts_tablet_manager.cc:1918] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:35:55.109668 8573 log.cc:1196] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/8cd982f07a24450da8ab156f8772ba99
I20260928 14:35:55.109963 8573 ts_tablet_manager.cc:1939] T 8cd982f07a24450da8ab156f8772ba99 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting consensus metadata
I20260928 14:35:55.110625 7974 catalog_manager.cc:4841] TS 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471): tablet 8cd982f07a24450da8ab156f8772ba99 (table test-table [id=25477b05c9f64141874498cbf6960267]) successfully deleted
Sep 28 14:35:55 dist-test-slave-06lh krb5kdc[7944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:35:55 dist-test-slave-06lh krb5kdc[7944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606155, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:55.136577 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.122265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.6.23.254:35779)
0928 14:35:55.122578 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:55.122589 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:55.122689 (+ 100us) server_negotiation.cc:407] Connection header received
0928 14:35:55.122817 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:55.122820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:55.122867 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:55.122952 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:55.123735 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.124290 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.125092 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.125309 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.125382 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.132600 (+ 7218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:55.132604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:55.132606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:55.132638 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:55.134431 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:55.135080 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:55.135084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:55.135085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:55.135178 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:55.135605 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:55.135608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:55.135609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:55.135821 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:55.135935 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:55.136198 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 14:35:55.136317 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260928 14:35:55.139424 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:35:55.139899 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:55.144773 8126 tablet_service.cc:1467] Processing CreateTablet for tablet 935d189024c84abfbb24b935e0ba531b (DEFAULT_TABLE table=test-table [id=5e16e572032449d6854acb313cdded39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:55.144822 8266 tablet_service.cc:1467] Processing CreateTablet for tablet 935d189024c84abfbb24b935e0ba531b (DEFAULT_TABLE table=test-table [id=5e16e572032449d6854acb313cdded39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:55.144888 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 935d189024c84abfbb24b935e0ba531b (DEFAULT_TABLE table=test-table [id=5e16e572032449d6854acb313cdded39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:55.145121 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935d189024c84abfbb24b935e0ba531b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:55.145123 8266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935d189024c84abfbb24b935e0ba531b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:55.145136 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935d189024c84abfbb24b935e0ba531b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:55.147737 8586 tablet_bootstrap.cc:492] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: Bootstrap starting.
I20260928 14:35:55.147769 8585 tablet_bootstrap.cc:492] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: Bootstrap starting.
I20260928 14:35:55.148065 8584 tablet_bootstrap.cc:492] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: Bootstrap starting.
I20260928 14:35:55.148633 8586 tablet_bootstrap.cc:654] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:55.148710 8585 tablet_bootstrap.cc:654] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:55.149035 8584 tablet_bootstrap.cc:654] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:55.149933 8585 tablet_bootstrap.cc:492] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: No bootstrap required, opened a new log
I20260928 14:35:55.150036 8585 ts_tablet_manager.cc:1397] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:55.150370 8584 tablet_bootstrap.cc:492] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: No bootstrap required, opened a new log
I20260928 14:35:55.150424 8586 tablet_bootstrap.cc:492] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: No bootstrap required, opened a new log
I20260928 14:35:55.150467 8584 ts_tablet_manager.cc:1397] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:35:55.150504 8586 ts_tablet_manager.cc:1397] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:35:55.150532 8585 raft_consensus.cc:348] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.150642 8585 raft_consensus.cc:374] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:55.150677 8585 raft_consensus.cc:729] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c33655f9eb4e47af99ded706a370f9, State: Initialized, Role: FOLLOWER
I20260928 14:35:55.150794 8585 consensus_queue.cc:260] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.151013 8586 raft_consensus.cc:348] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.151122 8586 raft_consensus.cc:374] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:55.151153 8586 raft_consensus.cc:729] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bd86ab9e6640e6bdb348e331880ae3, State: Initialized, Role: FOLLOWER
I20260928 14:35:55.151187 8585 ts_tablet_manager.cc:1428] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:55.151124 8584 raft_consensus.cc:348] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.151320 8584 raft_consensus.cc:374] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:55.151357 8584 raft_consensus.cc:729] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a762fc6629d414e8e61f52f6e39f5b3, State: Initialized, Role: FOLLOWER
I20260928 14:35:55.151335 8586 consensus_queue.cc:260] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.151598 8586 ts_tablet_manager.cc:1428] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:35:55.151976 8584 consensus_queue.cc:260] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.152415 8584 ts_tablet_manager.cc:1428] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:55.289062 8590 raft_consensus.cc:482] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:55.289214 8590 raft_consensus.cc:504] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.290306 8590 leader_election.cc:288] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103), 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471)
I20260928 14:35:55.295099 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.290621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37471 (local address 127.6.23.194:52395)
0928 14:35:55.290951 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:55.290968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:55.291084 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:55.291472 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:55.291476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:55.291496 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:55.291658 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:55.291665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.292683 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.292686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:55.293708 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:55.293716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.294595 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.294599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:55.294718 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.294746 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 14:35:55.294834 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:35:55.294908 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"thread_start_us":143,"threads_started":1}
I20260928 14:35:55.295262 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.290724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:52395 (local address 127.6.23.195:37471)
0928 14:35:55.291027 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:55.291036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:55.291054 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:35:55.291241 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:55.291246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:55.291312 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:55.291404 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:55.291775 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.292583 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.293830 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.294463 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.294561 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.294604 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:55.295008 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 14:35:55.295089 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
I20260928 14:35:55.295670 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.290690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:54503 (local address 127.6.23.193:41103)
0928 14:35:55.291069 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:55.291077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:55.291359 (+ 282us) server_negotiation.cc:407] Connection header received
0928 14:35:55.291502 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:55.291505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:55.291553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:55.291643 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:35:55.292380 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.293065 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.294178 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.294750 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.294872 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.294933 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:55.295453 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 14:35:55.295532 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":125,"threads_started":1}
I20260928 14:35:55.295689 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935d189024c84abfbb24b935e0ba531b" candidate_uuid: "14c33655f9eb4e47af99ded706a370f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bd86ab9e6640e6bdb348e331880ae3" is_pre_election: true
I20260928 14:35:55.295876 8425 raft_consensus.cc:2454] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c33655f9eb4e47af99ded706a370f9 in term 0.
I20260928 14:35:55.295961 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.290624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41103 (local address 127.6.23.194:54503)
0928 14:35:55.291263 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:55.291277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:55.291388 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:55.291660 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:55.291662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:55.291676 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:35:55.292260 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:55.292270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.293176 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.293178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:55.294054 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:55.294064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.295018 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.295021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:55.295123 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.295147 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:35:55.295221 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:35:55.295320 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":251,"thread_start_us":76,"threads_started":1}
I20260928 14:35:55.296350 8218 leader_election.cc:302] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c33655f9eb4e47af99ded706a370f9, 45bd86ab9e6640e6bdb348e331880ae3; no voters:
I20260928 14:35:55.296437 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935d189024c84abfbb24b935e0ba531b" candidate_uuid: "14c33655f9eb4e47af99ded706a370f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" is_pre_election: true
I20260928 14:35:55.296605 8146 raft_consensus.cc:2454] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c33655f9eb4e47af99ded706a370f9 in term 0.
I20260928 14:35:55.296663 8590 raft_consensus.cc:2781] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:55.296720 8590 raft_consensus.cc:482] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:55.296774 8590 raft_consensus.cc:3037] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:55.297973 8590 raft_consensus.cc:504] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.298373 8590 leader_election.cc:288] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [CANDIDATE]: Term 1 election: Requested vote from peers 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103), 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471)
I20260928 14:35:55.298681 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935d189024c84abfbb24b935e0ba531b" candidate_uuid: "14c33655f9eb4e47af99ded706a370f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a762fc6629d414e8e61f52f6e39f5b3"
I20260928 14:35:55.298825 8146 raft_consensus.cc:3037] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:55.298859 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935d189024c84abfbb24b935e0ba531b" candidate_uuid: "14c33655f9eb4e47af99ded706a370f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bd86ab9e6640e6bdb348e331880ae3"
I20260928 14:35:55.299026 8425 raft_consensus.cc:3037] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:55.299873 8146 raft_consensus.cc:2454] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c33655f9eb4e47af99ded706a370f9 in term 1.
I20260928 14:35:55.300034 8425 raft_consensus.cc:2454] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c33655f9eb4e47af99ded706a370f9 in term 1.
I20260928 14:35:55.300177 8219 leader_election.cc:302] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c33655f9eb4e47af99ded706a370f9, 3a762fc6629d414e8e61f52f6e39f5b3; no voters:
I20260928 14:35:55.300446 8590 raft_consensus.cc:2781] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:55.300671 8590 raft_consensus.cc:686] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 1 LEADER]: Becoming Leader. State: Replica: 14c33655f9eb4e47af99ded706a370f9, State: Running, Role: LEADER
I20260928 14:35:55.300931 8590 consensus_queue.cc:237] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.303556 7989 catalog_manager.cc:5495] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194). New cstate: current_term: 1 leader_uuid: "14c33655f9eb4e47af99ded706a370f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } health_report { overall_health: UNKNOWN } } }
I20260928 14:35:55.349035 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.345410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.6.23.194:45915)
0928 14:35:55.345703 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:55.345709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:55.345721 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:35:55.345794 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:55.345796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:55.345854 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:55.345961 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:55.346170 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.346728 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.347588 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.347817 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.347898 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.348121 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:55.348612 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:55.348756 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:55.348791 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 14:35:55.348837 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":140,"threads_started":1}
I20260928 14:35:55.359210 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.355997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:37733 (local address 127.6.23.195:37471)
0928 14:35:55.356123 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:55.356131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:55.356190 (+ 59us) server_negotiation.cc:407] Connection header received
0928 14:35:55.356350 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:55.356353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:55.356413 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:55.356487 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:55.356741 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.357244 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.358122 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.358344 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.358461 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.358752 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:55.358828 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:55.358908 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:55.359002 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 14:35:55.359055 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:35:55.359243 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.355786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37471 (local address 127.6.23.194:37733)
0928 14:35:55.356120 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:55.356136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:55.356241 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:55.356507 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:55.356510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:55.356518 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:55.356641 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:55.356646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.357351 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.357357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:55.357996 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:55.358004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.358473 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.358476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:55.358579 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.358621 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:55.358922 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:55.358930 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:35:55.358991 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 14:35:55.359073 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260928 14:35:55.365533 8425 raft_consensus.cc:1261] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Refusing update from remote peer 14c33655f9eb4e47af99ded706a370f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:55.365638 8146 raft_consensus.cc:1261] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Refusing update from remote peer 14c33655f9eb4e47af99ded706a370f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:55.366048 8596 consensus_queue.cc:1035] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:55.366202 8590 consensus_queue.cc:1035] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:55.383536 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.379987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:45367 (local address 127.6.23.194:45915)
0928 14:35:55.380088 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:55.380093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:55.380446 (+ 353us) server_negotiation.cc:407] Connection header received
0928 14:35:55.380567 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:55.380570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:55.380615 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:55.380701 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:35:55.380940 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.381461 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.382406 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.382668 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.382782 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.383045 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:35:55.383166 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:35:55.383263 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:55.383340 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:35:55.383388 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 14:35:55.383570 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:55.379930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:45915 (local address 127.6.23.195:45367)
0928 14:35:55.380345 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:55.380356 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:55.380482 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:55.380710 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:55.380713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:55.380720 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:35:55.380826 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:55.380831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.381570 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.381575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:55.382278 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:55.382286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:55.382790 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:55.382794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:55.382881 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:55.382917 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:35:55.383261 (+ 344us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:35:55.383268 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:35:55.383333 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:35:55.383414 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":109,"threads_started":1}
I20260928 14:35:55.405622 7988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:55.405807 7988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:55.408041 7988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: Sending DeleteTablet for 3 replicas of tablet 935d189024c84abfbb24b935e0ba531b
I20260928 14:35:55.408658 8266 tablet_service.cc:1514] Processing DeleteTablet for tablet 935d189024c84abfbb24b935e0ba531b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34428
I20260928 14:35:55.408658 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet 935d189024c84abfbb24b935e0ba531b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:56376
I20260928 14:35:55.408843 8574 tablet_replica.cc:330] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: stopping tablet replica
I20260928 14:35:55.408843 8575 tablet_replica.cc:330] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: stopping tablet replica
I20260928 14:35:55.408820 8405 tablet_service.cc:1514] Processing DeleteTablet for tablet 935d189024c84abfbb24b935e0ba531b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:45678
I20260928 14:35:55.408952 8575 raft_consensus.cc:2229] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:35:55.408980 8574 raft_consensus.cc:2229] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:55.409008 8573 tablet_replica.cc:330] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: stopping tablet replica
I20260928 14:35:55.409087 8574 raft_consensus.cc:2258] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:55.409118 8573 raft_consensus.cc:2229] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:55.409227 8573 raft_consensus.cc:2258] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:55.409260 8575 raft_consensus.cc:2258] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:55.410053 8575 ts_tablet_manager.cc:1905] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:55.410053 8573 ts_tablet_manager.cc:1905] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:55.410383 8574 ts_tablet_manager.cc:1905] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:55.412078 7988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:35:55.412540 7988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:35:55.413138 8574 ts_tablet_manager.cc:1918] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:55.413228 8574 log.cc:1196] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/935d189024c84abfbb24b935e0ba531b
I20260928 14:35:55.412711 8573 ts_tablet_manager.cc:1918] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:55.413419 8573 log.cc:1196] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/935d189024c84abfbb24b935e0ba531b
I20260928 14:35:55.413501 8574 ts_tablet_manager.cc:1939] T 935d189024c84abfbb24b935e0ba531b P 3a762fc6629d414e8e61f52f6e39f5b3: Deleting consensus metadata
I20260928 14:35:55.414275 7975 catalog_manager.cc:4841] TS 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103): tablet 935d189024c84abfbb24b935e0ba531b (table test-table [id=5e16e572032449d6854acb313cdded39]) successfully deleted
I20260928 14:35:55.415305 8575 ts_tablet_manager.cc:1918] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:35:55.415441 8575 log.cc:1196] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/935d189024c84abfbb24b935e0ba531b
I20260928 14:35:55.415704 8575 ts_tablet_manager.cc:1939] T 935d189024c84abfbb24b935e0ba531b P 14c33655f9eb4e47af99ded706a370f9: Deleting consensus metadata
I20260928 14:35:55.415903 8573 ts_tablet_manager.cc:1939] T 935d189024c84abfbb24b935e0ba531b P 45bd86ab9e6640e6bdb348e331880ae3: Deleting consensus metadata
I20260928 14:35:55.416378 7973 catalog_manager.cc:4841] TS 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915): tablet 935d189024c84abfbb24b935e0ba531b (table test-table [id=5e16e572032449d6854acb313cdded39]) successfully deleted
I20260928 14:35:55.416644 7974 catalog_manager.cc:4841] TS 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471): tablet 935d189024c84abfbb24b935e0ba531b (table test-table [id=5e16e572032449d6854acb313cdded39]) successfully deleted
I20260928 14:35:55.417829 8266 tablet_service.cc:1467] Processing CreateTablet for tablet 25623bb8649a45589b488a6869824ce9 (DEFAULT_TABLE table=test-table [id=6afe66984cfb4f948abaf92ba9711583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:55.417953 8405 tablet_service.cc:1467] Processing CreateTablet for tablet 25623bb8649a45589b488a6869824ce9 (DEFAULT_TABLE table=test-table [id=6afe66984cfb4f948abaf92ba9711583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:55.418138 8266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25623bb8649a45589b488a6869824ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:55.418210 8405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25623bb8649a45589b488a6869824ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:55.419662 8126 tablet_service.cc:1467] Processing CreateTablet for tablet 25623bb8649a45589b488a6869824ce9 (DEFAULT_TABLE table=test-table [id=6afe66984cfb4f948abaf92ba9711583]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:35:55.419927 8126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25623bb8649a45589b488a6869824ce9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:55.420203 8585 tablet_bootstrap.cc:492] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: Bootstrap starting.
I20260928 14:35:55.420588 8586 tablet_bootstrap.cc:492] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: Bootstrap starting.
I20260928 14:35:55.421140 8585 tablet_bootstrap.cc:654] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:55.421361 8586 tablet_bootstrap.cc:654] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:55.422268 8584 tablet_bootstrap.cc:492] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3: Bootstrap starting.
I20260928 14:35:55.422645 8586 tablet_bootstrap.cc:492] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: No bootstrap required, opened a new log
I20260928 14:35:55.422727 8586 ts_tablet_manager.cc:1397] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:35:55.423122 8584 tablet_bootstrap.cc:654] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:55.423089 8586 raft_consensus.cc:348] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.423192 8586 raft_consensus.cc:374] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:55.423242 8586 raft_consensus.cc:729] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45bd86ab9e6640e6bdb348e331880ae3, State: Initialized, Role: FOLLOWER
I20260928 14:35:55.423357 8586 consensus_queue.cc:260] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.423630 8586 ts_tablet_manager.cc:1428] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:55.424021 8585 tablet_bootstrap.cc:492] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: No bootstrap required, opened a new log
I20260928 14:35:55.424104 8585 ts_tablet_manager.cc:1397] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:35:55.424539 8584 tablet_bootstrap.cc:492] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3: No bootstrap required, opened a new log
I20260928 14:35:55.424629 8584 ts_tablet_manager.cc:1397] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:35:55.424561 8585 raft_consensus.cc:348] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.424676 8585 raft_consensus.cc:374] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:55.424715 8585 raft_consensus.cc:729] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c33655f9eb4e47af99ded706a370f9, State: Initialized, Role: FOLLOWER
I20260928 14:35:55.424844 8585 consensus_queue.cc:260] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.424988 8584 raft_consensus.cc:348] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.425086 8584 raft_consensus.cc:374] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:55.425115 8584 raft_consensus.cc:729] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a762fc6629d414e8e61f52f6e39f5b3, State: Initialized, Role: FOLLOWER
I20260928 14:35:55.425145 8585 ts_tablet_manager.cc:1428] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:35:55.425225 8584 consensus_queue.cc:260] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.425459 8584 ts_tablet_manager.cc:1428] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:35:55.607465 8590 raft_consensus.cc:482] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:35:55.607637 8590 raft_consensus.cc:504] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.608103 8590 leader_election.cc:288] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103), 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471)
I20260928 14:35:55.608593 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25623bb8649a45589b488a6869824ce9" candidate_uuid: "14c33655f9eb4e47af99ded706a370f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bd86ab9e6640e6bdb348e331880ae3" is_pre_election: true
I20260928 14:35:55.608593 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25623bb8649a45589b488a6869824ce9" candidate_uuid: "14c33655f9eb4e47af99ded706a370f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" is_pre_election: true
I20260928 14:35:55.608752 8146 raft_consensus.cc:2454] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c33655f9eb4e47af99ded706a370f9 in term 0.
I20260928 14:35:55.608752 8425 raft_consensus.cc:2454] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c33655f9eb4e47af99ded706a370f9 in term 0.
I20260928 14:35:55.609071 8219 leader_election.cc:302] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c33655f9eb4e47af99ded706a370f9, 3a762fc6629d414e8e61f52f6e39f5b3; no voters:
I20260928 14:35:55.609261 8590 raft_consensus.cc:2781] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:35:55.609318 8590 raft_consensus.cc:482] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:35:55.609344 8590 raft_consensus.cc:3037] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:55.610733 8590 raft_consensus.cc:504] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.611102 8590 leader_election.cc:288] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [CANDIDATE]: Term 1 election: Requested vote from peers 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103), 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471)
I20260928 14:35:55.611447 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25623bb8649a45589b488a6869824ce9" candidate_uuid: "14c33655f9eb4e47af99ded706a370f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a762fc6629d414e8e61f52f6e39f5b3"
I20260928 14:35:55.611496 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25623bb8649a45589b488a6869824ce9" candidate_uuid: "14c33655f9eb4e47af99ded706a370f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45bd86ab9e6640e6bdb348e331880ae3"
I20260928 14:35:55.611588 8146 raft_consensus.cc:3037] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:55.611619 8425 raft_consensus.cc:3037] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:55.612790 8146 raft_consensus.cc:2454] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c33655f9eb4e47af99ded706a370f9 in term 1.
I20260928 14:35:55.612790 8425 raft_consensus.cc:2454] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c33655f9eb4e47af99ded706a370f9 in term 1.
I20260928 14:35:55.613092 8219 leader_election.cc:302] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c33655f9eb4e47af99ded706a370f9, 3a762fc6629d414e8e61f52f6e39f5b3; no voters:
I20260928 14:35:55.613263 8590 raft_consensus.cc:2781] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:55.613359 8590 raft_consensus.cc:686] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 1 LEADER]: Becoming Leader. State: Replica: 14c33655f9eb4e47af99ded706a370f9, State: Running, Role: LEADER
I20260928 14:35:55.613467 8590 consensus_queue.cc:237] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } }
I20260928 14:35:55.615101 7988 catalog_manager.cc:5495] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194). New cstate: current_term: 1 leader_uuid: "14c33655f9eb4e47af99ded706a370f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c33655f9eb4e47af99ded706a370f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 45915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 } health_report { overall_health: UNKNOWN } } }
I20260928 14:35:55.670899 8425 raft_consensus.cc:1261] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Refusing update from remote peer 14c33655f9eb4e47af99ded706a370f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:55.670944 8146 raft_consensus.cc:1261] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Refusing update from remote peer 14c33655f9eb4e47af99ded706a370f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:35:55.671363 8596 consensus_queue.cc:1035] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45bd86ab9e6640e6bdb348e331880ae3" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:35:55.671545 8590 consensus_queue.cc:1035] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a762fc6629d414e8e61f52f6e39f5b3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:35:55.688599 8557 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:35:57.240476 7989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:57.240626 7989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:35:57.242935 7989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 374a8da70b2c44538b81461cae778c82: Sending DeleteTablet for 3 replicas of tablet 25623bb8649a45589b488a6869824ce9
I20260928 14:35:57.243546 8126 tablet_service.cc:1514] Processing DeleteTablet for tablet 25623bb8649a45589b488a6869824ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:57 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:56376
I20260928 14:35:57.243613 8406 tablet_service.cc:1514] Processing DeleteTablet for tablet 25623bb8649a45589b488a6869824ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:57 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:45678
I20260928 14:35:57.243920 8266 tablet_service.cc:1514] Processing DeleteTablet for tablet 25623bb8649a45589b488a6869824ce9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:35:57 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34428
I20260928 14:35:57.243978 8638 tablet_replica.cc:330] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3: stopping tablet replica
I20260928 14:35:57.243981 8639 tablet_replica.cc:330] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: stopping tablet replica
I20260928 14:35:57.244115 8638 raft_consensus.cc:2229] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:57.244124 8639 raft_consensus.cc:2229] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:35:57.244239 8639 raft_consensus.cc:2258] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:57.244261 8638 raft_consensus.cc:2258] T 25623bb8649a45589b488a6869824ce9 P 3a762fc6629d414e8e61f52f6e39f5b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:57.244445 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 8061
I20260928 14:35:57.246691 8639 ts_tablet_manager.cc:1905] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:57.247402 8640 tablet_replica.cc:330] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: stopping tablet replica
I20260928 14:35:57.247550 8640 raft_consensus.cc:2229] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:35:57.247766 8640 raft_consensus.cc:2258] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:35:57.248297 8640 ts_tablet_manager.cc:1905] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:35:57.250651 8639 ts_tablet_manager.cc:1918] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:35:57.250741 8639 log.cc:1196] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/25623bb8649a45589b488a6869824ce9
I20260928 14:35:57.250831 8640 ts_tablet_manager.cc:1918] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:35:57.250945 8640 log.cc:1196] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/25623bb8649a45589b488a6869824ce9
I20260928 14:35:57.251024 8639 ts_tablet_manager.cc:1939] T 25623bb8649a45589b488a6869824ce9 P 45bd86ab9e6640e6bdb348e331880ae3: Deleting consensus metadata
I20260928 14:35:57.251308 8640 ts_tablet_manager.cc:1939] T 25623bb8649a45589b488a6869824ce9 P 14c33655f9eb4e47af99ded706a370f9: Deleting consensus metadata
I20260928 14:35:57.252022 7974 catalog_manager.cc:4841] TS 45bd86ab9e6640e6bdb348e331880ae3 (127.6.23.195:37471): tablet 25623bb8649a45589b488a6869824ce9 (table test-table [id=6afe66984cfb4f948abaf92ba9711583]) successfully deleted
I20260928 14:35:57.252022 7973 catalog_manager.cc:4841] TS 14c33655f9eb4e47af99ded706a370f9 (127.6.23.194:45915): tablet 25623bb8649a45589b488a6869824ce9 (table test-table [id=6afe66984cfb4f948abaf92ba9711583]) successfully deleted
W20260928 14:35:57.253386 7975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:41103 (error 108)
I20260928 14:35:57.253566 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 8201
I20260928 14:35:57.254980 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:57.254582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41103 (local address 127.0.0.1:56380)
0928 14:35:57.254819 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:57.254894 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":99,"threads_started":1}
W20260928 14:35:57.255131 7975 catalog_manager.cc:4584] TS 3a762fc6629d414e8e61f52f6e39f5b3 (127.6.23.193:41103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25623bb8649a45589b488a6869824ce9: Network error: Client connection negotiation failed: client connection to 127.6.23.193:41103: connect: Connection refused (error 111)
I20260928 14:35:57.261675 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 8341
I20260928 14:35:57.268254 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 7957
2026-09-28T14:35:57Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6446 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:35:57 dist-test-slave-06lh krb5kdc[8645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:35:57 dist-test-slave-06lh krb5kdc[8645](info): set up 2 sockets
Sep 28 14:35:57 dist-test-slave-06lh krb5kdc[8645](info): commencing operation
krb5kdc: starting...
W20260928 14:35:58.319864 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:35:58 dist-test-slave-06lh krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:35:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:35:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:35:58.442332 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:46805
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:46805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:58.551982 8658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:58.552289 8658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:58.552377 8658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:58.556375 8658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:35:58.556454 8658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:58.556493 8658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:58.556551 8658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:35:58.556588 8658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:35:58.561962 8658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:46805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:46805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:58.563014 8658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:58.564043 8658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:58.573501 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:58.573519 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:58.573508 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:58.573951 8658 server_base.cc:1034] running on GCE node
I20260928 14:35:58.574476 8658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:58.575665 8658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:58.576845 8658 hybrid_clock.cc:648] HybridClock initialized: now 1790606158576819 us; error 37 us; skew 500 ppm
Sep 28 14:35:58 dist-test-slave-06lh krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606158, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:58.579753 8658 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:35:58.580603 8658 webserver.cc:460] Webserver started at http://127.6.23.254:37805/ using document root <none> and password file <none>
I20260928 14:35:58.581120 8658 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:58.581179 8658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:58.581341 8658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:58.583406 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "9a85d98d29874d3bb754735c969fa2cd"
format_stamp: "Formatted at 2026-09-28 14:35:58 on dist-test-slave-06lh"
server_key: "6c0df0bda4a60a443205db472298c6b0"
server_key_iv: "87c7544c9cdcb0011034955e7fd3911c"
server_key_version: "encryptionkey@0"
I20260928 14:35:58.583942 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "9a85d98d29874d3bb754735c969fa2cd"
format_stamp: "Formatted at 2026-09-28 14:35:58 on dist-test-slave-06lh"
server_key: "6c0df0bda4a60a443205db472298c6b0"
server_key_iv: "87c7544c9cdcb0011034955e7fd3911c"
server_key_version: "encryptionkey@0"
I20260928 14:35:58.587669 8658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:35:58.590036 8673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:58.591358 8658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:58.591498 8658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "9a85d98d29874d3bb754735c969fa2cd"
format_stamp: "Formatted at 2026-09-28 14:35:58 on dist-test-slave-06lh"
server_key: "6c0df0bda4a60a443205db472298c6b0"
server_key_iv: "87c7544c9cdcb0011034955e7fd3911c"
server_key_version: "encryptionkey@0"
I20260928 14:35:58.591622 8658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:58.621140 8658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:58.624315 8658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:58.624533 8658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:58.631783 8658 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:46805
I20260928 14:35:58.631803 8725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:46805 every 8 connection(s)
I20260928 14:35:58.632753 8658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:35:58.635617 8726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:35:58.639083 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 8658
I20260928 14:35:58.639204 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:35:58.639472 6239 external_mini_cluster.cc:1428] Setting key 4627da978e8c206e182ff16d08b2ec9a
I20260928 14:35:58.641615 8726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd: Bootstrap starting.
I20260928 14:35:58.644071 8726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd: Neither blocks nor log segments found. Creating new log.
I20260928 14:35:58.644897 8726 log.cc:824] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd: Log is configured to *not* fsync() on all Append() calls
I20260928 14:35:58.646788 8726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd: No bootstrap required, opened a new log
Sep 28 14:35:58 dist-test-slave-06lh krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:58.650768 8726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85d98d29874d3bb754735c969fa2cd" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 46805 } }
I20260928 14:35:58.651017 8726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:35:58.651100 8726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a85d98d29874d3bb754735c969fa2cd, State: Initialized, Role: FOLLOWER
I20260928 14:35:58.651625 8726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85d98d29874d3bb754735c969fa2cd" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 46805 } }
I20260928 14:35:58.651778 8726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:35:58.651870 8726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:35:58.651969 8726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:35:58.653043 8726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85d98d29874d3bb754735c969fa2cd" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 46805 } }
I20260928 14:35:58.653421 8726 leader_election.cc:302] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a85d98d29874d3bb754735c969fa2cd; no voters:
I20260928 14:35:58.653741 8726 leader_election.cc:288] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:35:58.653849 8731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:35:58.654017 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:58.640912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54820 (local address 127.6.23.254:46805)
0928 14:35:58.641357 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:58.641372 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:58.641409 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:35:58.642107 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:58.642125 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:58.642453 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:58.642795 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:58.643623 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:58.644705 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:58.645643 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:58.646124 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:58.646301 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:58.648826 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:58.648831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:58.648845 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:58.648872 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:58.650761 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:58.651294 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:58.651299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:58.651304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:58.651415 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:58.651712 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:58.651716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:58.651718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:58.652269 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:58.652491 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:58.652743 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 14:35:58.653041 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":144,"threads_started":1}
I20260928 14:35:58.654064 8731 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [term 1 LEADER]: Becoming Leader. State: Replica: 9a85d98d29874d3bb754735c969fa2cd, State: Running, Role: LEADER
I20260928 14:35:58.654744 8731 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85d98d29874d3bb754735c969fa2cd" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 46805 } }
I20260928 14:35:58.655125 8726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:35:58.656483 8732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a85d98d29874d3bb754735c969fa2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85d98d29874d3bb754735c969fa2cd" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 46805 } } }
I20260928 14:35:58.656606 8732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [sys.catalog]: This master's current role is: LEADER
I20260928 14:35:58.656985 8733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a85d98d29874d3bb754735c969fa2cd. Latest consensus state: current_term: 1 leader_uuid: "9a85d98d29874d3bb754735c969fa2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a85d98d29874d3bb754735c969fa2cd" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 46805 } } }
I20260928 14:35:58.657104 8733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd [sys.catalog]: This master's current role is: LEADER
W20260928 14:35:58.660175 8746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:35:58.660244 8746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:35:58.660326 8741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:35:58.661031 8741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:35:58.666374 8741 catalog_manager.cc:1347] Generated new cluster ID: fb2896c11c5a424db451342d3b93fcef
I20260928 14:35:58.666442 8741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:35:58.673017 8741 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:35:58.674041 8741 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:35:58.684346 8741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a85d98d29874d3bb754735c969fa2cd: Generated new TSK 0
I20260928 14:35:58.685523 8741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:35:58.720526 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:46805
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:58.832109 8752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:58.832424 8752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:58.832508 8752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:58.836526 8752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:58.836625 8752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:58.836723 8752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:35:58.842026 8752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:46805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:58.843179 8752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:58.844274 8752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:58.854041 8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:58.854117 8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:58.854063 8752 server_base.cc:1034] running on GCE node
W20260928 14:35:58.854138 8760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:58.854660 8752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:58.855437 8752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:58.856629 8752 hybrid_clock.cc:648] HybridClock initialized: now 1790606158856608 us; error 33 us; skew 500 ppm
Sep 28 14:35:58 dist-test-slave-06lh krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606158, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:58.859508 8752 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:35:58.860373 8752 webserver.cc:460] Webserver started at http://127.6.23.193:40793/ using document root <none> and password file <none>
I20260928 14:35:58.860945 8752 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:58.861013 8752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:58.861249 8752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:58.863341 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "048d886a1e0b48d1bedd5382108d9f74"
format_stamp: "Formatted at 2026-09-28 14:35:58 on dist-test-slave-06lh"
server_key: "c1115c76274b147b27156c4a2abb7cfc"
server_key_iv: "727f47d97e0c166a712fc7fc47528d6e"
server_key_version: "encryptionkey@0"
I20260928 14:35:58.863900 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "048d886a1e0b48d1bedd5382108d9f74"
format_stamp: "Formatted at 2026-09-28 14:35:58 on dist-test-slave-06lh"
server_key: "c1115c76274b147b27156c4a2abb7cfc"
server_key_iv: "727f47d97e0c166a712fc7fc47528d6e"
server_key_version: "encryptionkey@0"
I20260928 14:35:58.867637 8752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:35:58.869988 8767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:58.871174 8752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:35:58.871337 8752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "048d886a1e0b48d1bedd5382108d9f74"
format_stamp: "Formatted at 2026-09-28 14:35:58 on dist-test-slave-06lh"
server_key: "c1115c76274b147b27156c4a2abb7cfc"
server_key_iv: "727f47d97e0c166a712fc7fc47528d6e"
server_key_version: "encryptionkey@0"
I20260928 14:35:58.871429 8752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:58.885824 8752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:58.888723 8752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:58.888875 8752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:58.889426 8752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:58.890206 8752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:58.890252 8752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:58.890293 8752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:58.890308 8752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:58.899374 8752 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:32771
I20260928 14:35:58.899400 8880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:32771 every 8 connection(s)
I20260928 14:35:58.900296 8752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:35:58.907119 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 8752
I20260928 14:35:58.907217 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:35:58.907549 6239 external_mini_cluster.cc:1428] Setting key eb3b765c0d613e510d3f4660009156d6
Sep 28 14:35:58 dist-test-slave-06lh krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606158, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:58.915359 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:58.902158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:37627 (local address 127.6.23.254:46805)
0928 14:35:58.902312 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:58.902317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:58.903214 (+ 897us) server_negotiation.cc:407] Connection header received
0928 14:35:58.904716 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:58.904719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:58.904766 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:58.904842 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:58.906114 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:58.906641 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:58.907942 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:58.908261 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:58.908365 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:58.911174 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:58.911181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:58.911185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:58.911215 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:58.912903 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:58.913514 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:58.913521 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:58.913522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:58.913657 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:58.914032 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:58.914038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:58.914043 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:58.914325 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:58.914442 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:58.914931 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 14:35:58.915089 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:35:58.916234 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:58.902603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:46805 (local address 127.6.23.193:37627)
0928 14:35:58.903062 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:58.903105 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:58.904472 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:58.905049 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:58.905058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:58.905493 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:58.905975 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:58.905988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:58.906753 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:58.906756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:58.907817 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:58.907826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:58.908425 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:58.908429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:58.908608 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:58.909298 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:58.909314 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:58.910996 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:58.913034 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:58.913039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:58.913052 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:58.913392 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:58.913788 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:58.913790 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:58.913792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:58.913926 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:58.914435 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:58.914440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:58.914686 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 14:35:58.914846 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 14:35:58.915104 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:35:58.917529 8881 heartbeater.cc:347] Connected to a master server at 127.6.23.254:46805
I20260928 14:35:58.917783 8881 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:58.918287 8881 heartbeater.cc:511] Master 127.6.23.254:46805 requested a full tablet report, sending...
I20260928 14:35:58.919930 8690 ts_manager.cc:194] Registered new tserver with Master: 048d886a1e0b48d1bedd5382108d9f74 (127.6.23.193:32771)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:35:58.921586 8690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:37627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:35:58.934684 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:46805
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:59.046675 8886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:59.047003 8886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:59.047082 8886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:59.051182 8886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:59.051287 8886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:59.051410 8886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:35:59.056871 8886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:46805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:59.058002 8886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:59.059031 8886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:59.068701 8892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:59.068691 8894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:59.068691 8891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:59.069186 8886 server_base.cc:1034] running on GCE node
I20260928 14:35:59.069586 8886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:59.070274 8886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:59.071457 8886 hybrid_clock.cc:648] HybridClock initialized: now 1790606159071441 us; error 48 us; skew 500 ppm
Sep 28 14:35:59 dist-test-slave-06lh krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606159, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:59.074302 8886 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:35:59.075367 8886 webserver.cc:460] Webserver started at http://127.6.23.194:42609/ using document root <none> and password file <none>
I20260928 14:35:59.075951 8886 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:59.076050 8886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:59.076278 8886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:59.078410 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "e673db7febfd4088b3fb4e55f2c38d08"
format_stamp: "Formatted at 2026-09-28 14:35:59 on dist-test-slave-06lh"
server_key: "d46d4c1384a058ddc6a875c3e94a2eed"
server_key_iv: "bc7bc125d1c0f847c4a85a0b72dabf01"
server_key_version: "encryptionkey@0"
I20260928 14:35:59.078923 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "e673db7febfd4088b3fb4e55f2c38d08"
format_stamp: "Formatted at 2026-09-28 14:35:59 on dist-test-slave-06lh"
server_key: "d46d4c1384a058ddc6a875c3e94a2eed"
server_key_iv: "bc7bc125d1c0f847c4a85a0b72dabf01"
server_key_version: "encryptionkey@0"
I20260928 14:35:59.082706 8886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 14:35:59.085161 8901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:59.086395 8886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:35:59.086524 8886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "e673db7febfd4088b3fb4e55f2c38d08"
format_stamp: "Formatted at 2026-09-28 14:35:59 on dist-test-slave-06lh"
server_key: "d46d4c1384a058ddc6a875c3e94a2eed"
server_key_iv: "bc7bc125d1c0f847c4a85a0b72dabf01"
server_key_version: "encryptionkey@0"
I20260928 14:35:59.086634 8886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:59.110481 8886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:59.113663 8886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:59.113862 8886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:59.114470 8886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:59.115346 8886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:59.115393 8886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:59.115433 8886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:59.115484 8886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:59.124315 8886 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:37249
I20260928 14:35:59.124337 9014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:37249 every 8 connection(s)
I20260928 14:35:59.125218 8886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:35:59.131454 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 8886
I20260928 14:35:59.131577 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:35:59.131837 6239 external_mini_cluster.cc:1428] Setting key fe476639ae8a72f7ec825fe9c36004c7
Sep 28 14:35:59 dist-test-slave-06lh krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606159, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:59.139683 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:59.127052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:49795 (local address 127.6.23.254:46805)
0928 14:35:59.127192 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:59.127195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:59.128134 (+ 939us) server_negotiation.cc:407] Connection header received
0928 14:35:59.129493 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:59.129496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:59.129554 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:59.129636 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:59.130816 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:59.131426 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:59.132807 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:59.133096 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:59.133197 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:59.135781 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:59.135785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:59.135789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:59.135824 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:59.137554 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:59.138070 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:59.138075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:59.138077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:59.138161 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:59.138488 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:59.138493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:59.138494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:59.138713 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:59.138828 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:59.139373 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 14:35:59.139517 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:35:59.140563 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:59.127535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:46805 (local address 127.6.23.194:49795)
0928 14:35:59.127981 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:59.128024 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:59.129273 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:59.129809 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:59.129817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:59.130176 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:59.130664 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:59.130678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:59.131559 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:59.131583 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:59.132689 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:59.132697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:59.133266 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:59.133271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:59.133450 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:59.134079 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:59.134096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:59.135596 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:59.137675 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:59.137681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:59.137695 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:59.137961 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:59.138246 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:59.138248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:59.138250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:59.138380 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:59.138826 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:59.138834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:59.139100 (+ 266us) client_negotiation.cc:769] Sending connection context
0928 14:35:59.139332 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 14:35:59.139699 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:35:59.141892 9015 heartbeater.cc:347] Connected to a master server at 127.6.23.254:46805
I20260928 14:35:59.142148 9015 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:59.142685 9015 heartbeater.cc:511] Master 127.6.23.254:46805 requested a full tablet report, sending...
I20260928 14:35:59.143829 8690 ts_manager.cc:194] Registered new tserver with Master: e673db7febfd4088b3fb4e55f2c38d08 (127.6.23.194:37249)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:35:59.144541 8690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:49795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:35:59.158658 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:46805
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:35:59.268111 9020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:35:59.268392 9020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:35:59.268465 9020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:35:59.272426 9020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:35:59.272508 9020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:35:59.272627 9020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:35:59.277961 9020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:46805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:35:59.278978 9020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:35:59.279991 9020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:35:59.289196 9026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:59.289230 9028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:35:59.289249 9025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:35:59.289472 9020 server_base.cc:1034] running on GCE node
I20260928 14:35:59.289978 9020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:35:59.290640 9020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:35:59.291822 9020 hybrid_clock.cc:648] HybridClock initialized: now 1790606159291791 us; error 44 us; skew 500 ppm
Sep 28 14:35:59 dist-test-slave-06lh krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606159, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:35:59.294394 9020 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:35:59.295322 9020 webserver.cc:460] Webserver started at http://127.6.23.195:41625/ using document root <none> and password file <none>
I20260928 14:35:59.295871 9020 fs_manager.cc:359] Metadata directory not provided
I20260928 14:35:59.295958 9020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:35:59.296188 9020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:35:59.298184 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "6b8d30dd04c34f8b942106d01de72dbf"
format_stamp: "Formatted at 2026-09-28 14:35:59 on dist-test-slave-06lh"
server_key: "5e6967ae92714eaee1ffdfc1aba2908b"
server_key_iv: "30f7a29fd43478294dc5fce77d18a3b1"
server_key_version: "encryptionkey@0"
I20260928 14:35:59.298743 9020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "6b8d30dd04c34f8b942106d01de72dbf"
format_stamp: "Formatted at 2026-09-28 14:35:59 on dist-test-slave-06lh"
server_key: "5e6967ae92714eaee1ffdfc1aba2908b"
server_key_iv: "30f7a29fd43478294dc5fce77d18a3b1"
server_key_version: "encryptionkey@0"
I20260928 14:35:59.302413 9020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:35:59.304821 9035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:59.305963 9020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:35:59.306102 9020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "6b8d30dd04c34f8b942106d01de72dbf"
format_stamp: "Formatted at 2026-09-28 14:35:59 on dist-test-slave-06lh"
server_key: "5e6967ae92714eaee1ffdfc1aba2908b"
server_key_iv: "30f7a29fd43478294dc5fce77d18a3b1"
server_key_version: "encryptionkey@0"
I20260928 14:35:59.306210 9020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:35:59.325564 9020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:35:59.328445 9020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:35:59.328655 9020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:35:59.329234 9020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:35:59.330041 9020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:35:59.330111 9020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:59.330186 9020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:35:59.330231 9020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:35:59.339377 9020 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:41433
I20260928 14:35:59.339454 9148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:41433 every 8 connection(s)
I20260928 14:35:59.340281 9020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:35:59.345235 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 9020
I20260928 14:35:59.345350 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:35:59.345685 6239 external_mini_cluster.cc:1428] Setting key 74434d84b85b6484cbd5f5eb8188baa1
Sep 28 14:35:59 dist-test-slave-06lh krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606159, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:35:59.353801 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:59.342087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:38069 (local address 127.6.23.254:46805)
0928 14:35:59.342237 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:59.342246 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:59.343067 (+ 821us) server_negotiation.cc:407] Connection header received
0928 14:35:59.344414 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:59.344416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:59.344465 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:59.344542 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:59.346054 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:59.346836 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:59.348301 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:35:59.348516 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:35:59.348595 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:59.350852 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:35:59.350855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:35:59.350857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:35:59.350878 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:35:59.352135 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:59.352587 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:59.352589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:59.352591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:59.352659 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:35:59.352930 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:35:59.352932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:35:59.352934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:35:59.353088 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:35:59.353159 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:35:59.353593 (+ 434us) server_negotiation.cc:299] Negotiation successful
0928 14:35:59.353685 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:35:59.354594 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:35:59.342475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:46805 (local address 127.6.23.195:38069)
0928 14:35:59.342912 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 14:35:59.342954 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:35:59.344217 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:35:59.344718 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:35:59.344725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:35:59.345191 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:35:59.345886 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:59.345907 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:59.346998 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:59.347002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:59.348192 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:35:59.348201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:35:59.348638 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:35:59.348641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:35:59.348783 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:35:59.349301 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:35:59.349317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:35:59.350711 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:35:59.352232 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:59.352236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:59.352248 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:59.352503 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:59.352734 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:35:59.352736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:35:59.352737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:35:59.352857 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:35:59.353167 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:35:59.353172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:35:59.353381 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 14:35:59.353547 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 14:35:59.353778 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":118,"threads_started":1}
I20260928 14:35:59.355751 9149 heartbeater.cc:347] Connected to a master server at 127.6.23.254:46805
I20260928 14:35:59.356025 9149 heartbeater.cc:464] Registering TS with master...
I20260928 14:35:59.356593 9149 heartbeater.cc:511] Master 127.6.23.254:46805 requested a full tablet report, sending...
I20260928 14:35:59.357594 8690 ts_manager.cc:194] Registered new tserver with Master: 6b8d30dd04c34f8b942106d01de72dbf (127.6.23.195:41433)
I20260928 14:35:59.358124 8690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:38069
I20260928 14:35:59.359522 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:35:59.372391 8729 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:35:59.371000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54830 (local address 127.6.23.254:46805)
0928 14:35:59.371131 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:35:59.371134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:35:59.371343 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:35:59.371459 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:35:59.371461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:35:59.371508 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:35:59.371584 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:35:59.372233 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54830: BlockingRecv error: recv got EOF from 127.0.0.1:54830 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:35:59.373622 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 8752
I20260928 14:35:59.379356 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 8886
I20260928 14:35:59.384573 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 9020
I20260928 14:35:59.389704 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 8658
2026-09-28T14:35:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2119 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:35:59 dist-test-slave-06lh krb5kdc[9161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:35:59 dist-test-slave-06lh krb5kdc[9161](info): set up 2 sockets
Sep 28 14:35:59 dist-test-slave-06lh krb5kdc[9161](info): commencing operation
krb5kdc: starting...
W20260928 14:36:00.419775 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:00 dist-test-slave-06lh krb5kdc[9161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:36:00.540151 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40211
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:40211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:00.652105 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:00.652395 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:00.652436 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:00.656590 9174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:00.656656 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:00.656677 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:00.656693 9174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:00.656709 9174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:00.662293 9174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:40211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:00.663599 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:00.664724 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:00.673393 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:00.673383 9180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:00.673456 9174 server_base.cc:1034] running on GCE node
W20260928 14:36:00.673383 9182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:00.674234 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:00.675453 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:00.676658 9174 hybrid_clock.cc:648] HybridClock initialized: now 1790606160676629 us; error 43 us; skew 500 ppm
Sep 28 14:36:00 dist-test-slave-06lh krb5kdc[9161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606160, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:00.684116 9174 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:36:00.685359 9174 webserver.cc:460] Webserver started at http://127.6.23.254:46655/ using document root <none> and password file <none>
I20260928 14:36:00.686110 9174 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:00.686213 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:00.686445 9174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:00.689461 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "937c9c06112a41649fb357522d753efa"
format_stamp: "Formatted at 2026-09-28 14:36:00 on dist-test-slave-06lh"
server_key: "9a1553e049207f4dc961f161814a14d0"
server_key_iv: "2cdaf0217a0654f0d5afd8a9ee484ed9"
server_key_version: "encryptionkey@0"
I20260928 14:36:00.690167 9174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "937c9c06112a41649fb357522d753efa"
format_stamp: "Formatted at 2026-09-28 14:36:00 on dist-test-slave-06lh"
server_key: "9a1553e049207f4dc961f161814a14d0"
server_key_iv: "2cdaf0217a0654f0d5afd8a9ee484ed9"
server_key_version: "encryptionkey@0"
I20260928 14:36:00.694181 9174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:36:00.696790 9189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:00.698099 9174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:00.698263 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "937c9c06112a41649fb357522d753efa"
format_stamp: "Formatted at 2026-09-28 14:36:00 on dist-test-slave-06lh"
server_key: "9a1553e049207f4dc961f161814a14d0"
server_key_iv: "2cdaf0217a0654f0d5afd8a9ee484ed9"
server_key_version: "encryptionkey@0"
I20260928 14:36:00.698396 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:00.732398 9174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:00.735496 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:00.735710 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:00.743005 9174 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:40211
I20260928 14:36:00.743319 9241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:40211 every 8 connection(s)
I20260928 14:36:00.744076 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:00.746912 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 9174
I20260928 14:36:00.746956 9242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:00.747033 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:00.747386 6239 external_mini_cluster.cc:1428] Setting key b03f79ca630a5567e34bdb4bab603efa
I20260928 14:36:00.754118 9242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa: Bootstrap starting.
Sep 28 14:36:00 dist-test-slave-06lh krb5kdc[9161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:00.756970 9242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:00.757844 9242 log.cc:824] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:00.759924 9242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa: No bootstrap required, opened a new log
I20260928 14:36:00.761869 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:00.748738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.6.23.254:40211)
0928 14:36:00.749191 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:00.749205 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:00.749240 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:36:00.749854 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:00.749870 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:00.750221 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:00.750503 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:00.751438 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:00.752452 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:00.753635 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:00.754104 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:00.754252 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:00.757023 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:00.757029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:00.757043 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:00.757095 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:00.758992 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:00.759532 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:00.759537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:00.759542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:00.759655 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:00.759960 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:00.759963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:00.759964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:00.760405 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:00.760622 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:00.760867 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 14:36:00.761155 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":154,"threads_started":1}
I20260928 14:36:00.763839 9242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937c9c06112a41649fb357522d753efa" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40211 } }
I20260928 14:36:00.764106 9242 raft_consensus.cc:374] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:00.764196 9242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937c9c06112a41649fb357522d753efa, State: Initialized, Role: FOLLOWER
I20260928 14:36:00.764649 9242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937c9c06112a41649fb357522d753efa" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40211 } }
I20260928 14:36:00.764816 9242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:00.764912 9242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:00.764948 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:36:00.765053 9242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:00.765971 9242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937c9c06112a41649fb357522d753efa" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40211 } }
I20260928 14:36:00.766320 9242 leader_election.cc:302] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 937c9c06112a41649fb357522d753efa; no voters:
I20260928 14:36:00.766671 9242 leader_election.cc:288] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:00.766809 9248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:00.767058 9248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [term 1 LEADER]: Becoming Leader. State: Replica: 937c9c06112a41649fb357522d753efa, State: Running, Role: LEADER
I20260928 14:36:00.767455 9248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937c9c06112a41649fb357522d753efa" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40211 } }
I20260928 14:36:00.768240 9242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:00.769053 9249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "937c9c06112a41649fb357522d753efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937c9c06112a41649fb357522d753efa" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40211 } } }
I20260928 14:36:00.769176 9249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:00.769395 9249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 937c9c06112a41649fb357522d753efa. Latest consensus state: current_term: 1 leader_uuid: "937c9c06112a41649fb357522d753efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937c9c06112a41649fb357522d753efa" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40211 } } }
I20260928 14:36:00.769479 9249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:00.769886 9254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:00.773928 9254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:00.779820 9254 catalog_manager.cc:1347] Generated new cluster ID: c90966e5bce541618134e7803c5878ac
I20260928 14:36:00.779887 9254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:00.785061 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:00.780186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.6.23.254:40211)
0928 14:36:00.780320 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:00.780327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:00.780656 (+ 329us) server_negotiation.cc:407] Connection header received
0928 14:36:00.780819 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:00.780822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:00.780874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:00.780956 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:00.781473 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:00.782010 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:00.782852 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:00.783112 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:00.783224 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:00.784138 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:00.784142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:00.784145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:00.784600 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:00.784628 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:00.784645 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:00.784747 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:00.784851 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:36:00.784913 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:36:00.800608 9254 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:00.802073 9254 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:00.813455 9254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 937c9c06112a41649fb357522d753efa: Generated new TSK 0
I20260928 14:36:00.814216 9254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:36:00.830624 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 9174
2026-09-28T14:36:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1439 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:00 dist-test-slave-06lh krb5kdc[9275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:00 dist-test-slave-06lh krb5kdc[9275](info): set up 2 sockets
Sep 28 14:36:00 dist-test-slave-06lh krb5kdc[9275](info): commencing operation
krb5kdc: starting...
W20260928 14:36:01.871627 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:01 dist-test-slave-06lh krb5kdc[9275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:36:01.996387 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42807
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:42807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:02.108346 9288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:02.108672 9288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:02.108757 9288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:02.112857 9288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:02.112942 9288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:02.112980 9288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:02.113036 9288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:02.113077 9288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:02.118489 9288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:42807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:02.119689 9288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:02.120805 9288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:02.129558 9293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:02.129542 9294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:02.129577 9288 server_base.cc:1034] running on GCE node
W20260928 14:36:02.129541 9296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:02.130398 9288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:02.131646 9288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:02.132838 9288 hybrid_clock.cc:648] HybridClock initialized: now 1790606162132804 us; error 49 us; skew 500 ppm
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:02.135615 9288 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:36:02.136492 9288 webserver.cc:460] Webserver started at http://127.6.23.254:42745/ using document root <none> and password file <none>
I20260928 14:36:02.137074 9288 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:02.137166 9288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:02.137401 9288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:02.139556 9288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "5ed199c17ec644eb8b781afe89ca0a07"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "593bd8e42d07f004ea2fbb60c2636a66"
server_key_iv: "8d8dd0431850e3abc8582f84de35ae3e"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.140130 9288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "5ed199c17ec644eb8b781afe89ca0a07"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "593bd8e42d07f004ea2fbb60c2636a66"
server_key_iv: "8d8dd0431850e3abc8582f84de35ae3e"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.144280 9288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 14:36:02.146852 9303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.148195 9288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 14:36:02.148342 9288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "5ed199c17ec644eb8b781afe89ca0a07"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "593bd8e42d07f004ea2fbb60c2636a66"
server_key_iv: "8d8dd0431850e3abc8582f84de35ae3e"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.148464 9288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:02.167625 9288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:02.170557 9288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:02.170768 9288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:02.178256 9288 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:42807
I20260928 14:36:02.178555 9355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:42807 every 8 connection(s)
I20260928 14:36:02.179315 9288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:02.182129 9356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:02.182747 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 9288
I20260928 14:36:02.182884 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:02.183158 6239 external_mini_cluster.cc:1428] Setting key 7311f2ce072dda2ec005914ae849404c
I20260928 14:36:02.188645 9356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07: Bootstrap starting.
I20260928 14:36:02.191325 9356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07: Neither blocks nor log segments found. Creating new log.
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:02.192134 9356 log.cc:824] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:02.193948 9356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07: No bootstrap required, opened a new log
I20260928 14:36:02.197297 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:02.184560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.6.23.254:42807)
0928 14:36:02.184948 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:02.184964 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:02.184999 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:36:02.185658 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:02.185676 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:02.185999 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:02.186328 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:02.187195 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.188255 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.189359 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.189825 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.189946 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:02.192719 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:02.192724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:02.192737 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:02.192763 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:02.194519 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.194970 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.194981 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.194986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.195111 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.195443 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.195446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.195448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.195839 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:02.196058 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:02.196300 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 14:36:02.196590 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":122,"threads_started":1}
I20260928 14:36:02.197641 9356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed199c17ec644eb8b781afe89ca0a07" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42807 } }
I20260928 14:36:02.197889 9356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:02.197984 9356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed199c17ec644eb8b781afe89ca0a07, State: Initialized, Role: FOLLOWER
I20260928 14:36:02.198484 9356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed199c17ec644eb8b781afe89ca0a07" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42807 } }
I20260928 14:36:02.198648 9356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:02.198715 9356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:02.198807 9356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:02.199958 9356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed199c17ec644eb8b781afe89ca0a07" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42807 } }
I20260928 14:36:02.200250 9356 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ed199c17ec644eb8b781afe89ca0a07; no voters:
I20260928 14:36:02.200529 9356 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:02.200600 9361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:02.200796 9361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [term 1 LEADER]: Becoming Leader. State: Replica: 5ed199c17ec644eb8b781afe89ca0a07, State: Running, Role: LEADER
I20260928 14:36:02.201128 9361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed199c17ec644eb8b781afe89ca0a07" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42807 } }
I20260928 14:36:02.201790 9356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:02.202921 9363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ed199c17ec644eb8b781afe89ca0a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed199c17ec644eb8b781afe89ca0a07" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42807 } } }
I20260928 14:36:02.202960 9362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed199c17ec644eb8b781afe89ca0a07. Latest consensus state: current_term: 1 leader_uuid: "5ed199c17ec644eb8b781afe89ca0a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed199c17ec644eb8b781afe89ca0a07" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42807 } } }
I20260928 14:36:02.203109 9363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:02.203114 9362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:02.204147 9370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:02.206738 9370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:02.212416 9370 catalog_manager.cc:1347] Generated new cluster ID: 9ba97f1d9913429eb14d56db82c35dd2
I20260928 14:36:02.212491 9370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:02.225661 9370 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:02.226648 9370 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:02.233008 9370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ed199c17ec644eb8b781afe89ca0a07: Generated new TSK 0
I20260928 14:36:02.233723 9370 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:36:02.262758 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42807
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:02.377871 9382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:02.378242 9382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:02.378314 9382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:02.382437 9382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:02.382503 9382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:02.382603 9382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:02.388094 9382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:02.389231 9382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:02.390275 9382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:02.399896 9387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:02.399897 9388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:02.399916 9390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:02.400116 9382 server_base.cc:1034] running on GCE node
I20260928 14:36:02.400717 9382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:02.401405 9382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:02.402583 9382 hybrid_clock.cc:648] HybridClock initialized: now 1790606162402547 us; error 49 us; skew 500 ppm
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:02.405606 9382 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:36:02.406492 9382 webserver.cc:460] Webserver started at http://127.6.23.193:33445/ using document root <none> and password file <none>
I20260928 14:36:02.407060 9382 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:02.407153 9382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:02.407439 9382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:02.409518 9382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "dc84b294bcdf465ba862b4d6d4c089a8"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "359ccb7df9a381ed9041d222cd8061a2"
server_key_iv: "495998e914774c4b8e9d7e3aea85200e"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.410101 9382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "dc84b294bcdf465ba862b4d6d4c089a8"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "359ccb7df9a381ed9041d222cd8061a2"
server_key_iv: "495998e914774c4b8e9d7e3aea85200e"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.413895 9382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 14:36:02.416427 9397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.417755 9382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:02.417896 9382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "dc84b294bcdf465ba862b4d6d4c089a8"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "359ccb7df9a381ed9041d222cd8061a2"
server_key_iv: "495998e914774c4b8e9d7e3aea85200e"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.418030 9382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:02.428227 9382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:02.431547 9382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:02.431749 9382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:02.432337 9382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:02.433171 9382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:02.433243 9382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.433315 9382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:02.433358 9382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.442101 9382 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:35257
I20260928 14:36:02.442157 9510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:35257 every 8 connection(s)
I20260928 14:36:02.443053 9382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:02.449265 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 9382
I20260928 14:36:02.449364 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:02.449712 6239 external_mini_cluster.cc:1428] Setting key 1fb6e157d389abc7ba6bf808e7aa4b88
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:02.457953 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:02.444869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:36445 (local address 127.6.23.254:42807)
0928 14:36:02.445049 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:02.445054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:02.445991 (+ 937us) server_negotiation.cc:407] Connection header received
0928 14:36:02.447482 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:02.447485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:02.447535 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:02.447611 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:02.448768 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.449292 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.450853 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.451093 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.451207 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:02.453899 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:02.453903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:02.453905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:02.453930 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:02.455650 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.456235 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.456240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.456243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.456345 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.456689 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.456696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.456699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.456984 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:02.457099 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:02.457607 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 14:36:02.457759 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:36:02.458707 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:02.445314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42807 (local address 127.6.23.193:36445)
0928 14:36:02.445809 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:02.445866 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:02.447210 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:02.447776 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:02.447783 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:02.448156 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:02.448640 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:02.448653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.449454 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.449458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:02.450727 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:02.450737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.451271 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.451277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:02.451461 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:02.452130 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:02.452147 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:02.453730 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:02.455795 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:02.455802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:02.455821 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:02.456122 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:02.456443 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:02.456445 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:02.456447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:02.456588 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:02.457102 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:02.457108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:02.457349 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 14:36:02.457510 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 14:36:02.457775 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:36:02.460083 9511 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42807
I20260928 14:36:02.460353 9511 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:02.460865 9511 heartbeater.cc:511] Master 127.6.23.254:42807 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:36:02.462594 9320 ts_manager.cc:194] Registered new tserver with Master: dc84b294bcdf465ba862b4d6d4c089a8 (127.6.23.193:35257)
I20260928 14:36:02.464293 9320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:36445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:36:02.478827 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42807
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:02.592018 9516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:02.592423 9516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:02.592490 9516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:02.597263 9516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:02.597352 9516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:02.597441 9516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:02.603091 9516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:02.604362 9516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:02.605412 9516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:02.615490 9522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:02.615490 9521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:02.615655 9516 server_base.cc:1034] running on GCE node
W20260928 14:36:02.615553 9524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:02.616242 9516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:02.616941 9516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:02.618108 9516 hybrid_clock.cc:648] HybridClock initialized: now 1790606162618091 us; error 33 us; skew 500 ppm
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:02.620986 9516 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:36:02.621944 9516 webserver.cc:460] Webserver started at http://127.6.23.194:42845/ using document root <none> and password file <none>
I20260928 14:36:02.622547 9516 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:02.622609 9516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:02.622823 9516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:02.625098 9516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "bba8208bb77845e9b335f4510bf2ee78"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "20dbc0f151eb24ccd2ced49bbe053b33"
server_key_iv: "d5c506e796b47182e8e1da75b77ed11a"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.625677 9516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "bba8208bb77845e9b335f4510bf2ee78"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "20dbc0f151eb24ccd2ced49bbe053b33"
server_key_iv: "d5c506e796b47182e8e1da75b77ed11a"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.629545 9516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 14:36:02.632086 9531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.633319 9516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:02.633441 9516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "bba8208bb77845e9b335f4510bf2ee78"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "20dbc0f151eb24ccd2ced49bbe053b33"
server_key_iv: "d5c506e796b47182e8e1da75b77ed11a"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.633556 9516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:02.643052 9516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:02.646159 9516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:02.646370 9516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:02.646973 9516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:02.647887 9516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:02.647933 9516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.647996 9516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:02.648041 9516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.657770 9516 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:42967
I20260928 14:36:02.657790 9644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:42967 every 8 connection(s)
I20260928 14:36:02.658749 9516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:02.665274 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 9516
I20260928 14:36:02.665402 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:02.665704 6239 external_mini_cluster.cc:1428] Setting key 0af1eadb7bc10ee6f8e4feb1942f1119
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:02.674579 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:02.660699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:47421 (local address 127.6.23.254:42807)
0928 14:36:02.660850 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:02.660854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:02.661847 (+ 993us) server_negotiation.cc:407] Connection header received
0928 14:36:02.663322 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:02.663325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:02.663376 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:02.663450 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:02.664731 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.665434 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.666920 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.667263 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.667379 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:02.670272 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:02.670276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:02.670278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:02.670301 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:02.672094 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.672637 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.672644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.672648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.672764 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.673225 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.673231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.673235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.673533 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:02.673641 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:02.674216 (+ 575us) server_negotiation.cc:299] Negotiation successful
0928 14:36:02.674369 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:36:02.675418 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:02.661168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42807 (local address 127.6.23.194:47421)
0928 14:36:02.661682 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:02.661729 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:02.663039 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:02.663630 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:02.663637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:02.664022 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:02.664595 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:02.664610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.665579 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.665582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:02.666804 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:02.666814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.667434 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.667437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:02.667656 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:02.668451 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:02.668468 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:02.670083 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:02.672228 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:02.672234 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:02.672248 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:02.672525 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:02.672910 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:02.672913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:02.672915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:02.673112 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:02.673657 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:02.673662 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:02.673943 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 14:36:02.674121 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 14:36:02.674404 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:36:02.676834 9645 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42807
I20260928 14:36:02.677071 9645 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:02.677722 9645 heartbeater.cc:511] Master 127.6.23.254:42807 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:36:02.678900 9320 ts_manager.cc:194] Registered new tserver with Master: bba8208bb77845e9b335f4510bf2ee78 (127.6.23.194:42967)
I20260928 14:36:02.679963 9320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:47421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:36:02.698477 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42807
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:02.812994 9650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:02.813306 9650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:02.813390 9650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:02.817587 9650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:02.817679 9650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:02.817768 9650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:02.823987 9650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:02.825152 9650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:02.826251 9650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:02.837328 9658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:02.837422 9656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:02.837334 9655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:02.837878 9650 server_base.cc:1034] running on GCE node
I20260928 14:36:02.838357 9650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:02.839035 9650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:02.840235 9650 hybrid_clock.cc:648] HybridClock initialized: now 1790606162840194 us; error 62 us; skew 500 ppm
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:02.842971 9650 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:36:02.843948 9650 webserver.cc:460] Webserver started at http://127.6.23.195:40257/ using document root <none> and password file <none>
I20260928 14:36:02.844602 9650 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:02.844692 9650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:02.844929 9650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:02.847000 9650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "e5a592aae7e641759044e7dcb714cdb6"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "4734dd35d77114e9ee976444a1ca1774"
server_key_iv: "fedfc4537d2a3132b2f86d6989a449ea"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.847654 9650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "e5a592aae7e641759044e7dcb714cdb6"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "4734dd35d77114e9ee976444a1ca1774"
server_key_iv: "fedfc4537d2a3132b2f86d6989a449ea"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.852105 9650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:36:02.855656 9665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.857235 9650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:02.857366 9650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "e5a592aae7e641759044e7dcb714cdb6"
format_stamp: "Formatted at 2026-09-28 14:36:02 on dist-test-slave-06lh"
server_key: "4734dd35d77114e9ee976444a1ca1774"
server_key_iv: "fedfc4537d2a3132b2f86d6989a449ea"
server_key_version: "encryptionkey@0"
I20260928 14:36:02.857503 9650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:02.879000 9650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:02.882071 9650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:02.882282 9650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:02.882915 9650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:02.884016 9650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:02.884063 9650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.884130 9650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:02.884162 9650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:02.893195 9650 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:38629
I20260928 14:36:02.893217 9779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:38629 every 8 connection(s)
I20260928 14:36:02.894109 9650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:02.895392 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 9650
I20260928 14:36:02.895491 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:02.895784 6239 external_mini_cluster.cc:1428] Setting key 6d1ef71ffd5b3ec3c4bd4e6e8be03d5e
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:02.910944 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:02.896095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:51749 (local address 127.6.23.254:42807)
0928 14:36:02.896241 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:02.896252 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:02.897277 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 14:36:02.898946 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:02.898950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:02.899001 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:02.899127 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:02.900754 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.901354 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.902657 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.903003 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.903129 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:02.906739 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:02.906744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:02.906749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:02.906786 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:02.908787 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.909413 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.909435 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.909439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.909531 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.909959 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.909961 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.909962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.910149 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:02.910227 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:02.910714 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 14:36:02.910811 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:36:02.911862 9782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:02.896568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42807 (local address 127.6.23.195:51749)
0928 14:36:02.897131 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:02.897190 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:02.898610 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:02.899389 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:02.899399 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:02.899915 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:02.900611 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:02.900635 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.901469 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.901473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:02.902535 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:02.902545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.903215 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.903221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:02.903546 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:02.904290 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:02.904316 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:02.906512 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:02.908903 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:02.908912 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:02.908933 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:02.909305 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:02.909622 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:02.909624 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:02.909625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:02.909838 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:02.910240 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:02.910245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:02.910500 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 14:36:02.910665 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 14:36:02.910926 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":140,"threads_started":1}
I20260928 14:36:02.913004 9780 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42807
I20260928 14:36:02.913255 9780 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:02.913746 9780 heartbeater.cc:511] Master 127.6.23.254:42807 requested a full tablet report, sending...
I20260928 14:36:02.914736 9320 ts_manager.cc:194] Registered new tserver with Master: e5a592aae7e641759044e7dcb714cdb6 (127.6.23.195:38629)
I20260928 14:36:02.915337 9320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:51749
I20260928 14:36:02.921599 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 14:36:02 dist-test-slave-06lh krb5kdc[9275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606162, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:02.948237 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:02.939193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.6.23.254:42807)
0928 14:36:02.939387 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:02.939392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:02.939542 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:36:02.939727 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:02.939729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:02.939776 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:02.939863 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:02.940633 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.941186 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.942070 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:02.942299 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:02.942410 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:02.944988 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:02.944991 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:02.944993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:02.945020 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:02.946344 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.946855 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.946867 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.946869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.946952 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:02.947267 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:02.947274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:02.947276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:02.947539 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:02.947687 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:02.947930 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 14:36:02.948070 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 14:36:02.948670 9320 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36340
I20260928 14:36:02.950120 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 9382
I20260928 14:36:02.956233 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 9516
I20260928 14:36:02.963464 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 9650
I20260928 14:36:02.969278 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 9288
2026-09-28T14:36:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2140 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:03 dist-test-slave-06lh krb5kdc[9792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:03 dist-test-slave-06lh krb5kdc[9792](info): set up 2 sockets
Sep 28 14:36:03 dist-test-slave-06lh krb5kdc[9792](info): commencing operation
krb5kdc: starting...
W20260928 14:36:04.003867 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:04 dist-test-slave-06lh krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:36:04.127460 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40495
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:04.241050 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:04.241348 9805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:04.241389 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:04.245600 9805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:04.245669 9805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:04.245689 9805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:04.245705 9805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:04.245721 9805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:04.251391 9805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:40495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:04.252538 9805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:04.253675 9805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:04.262449 9811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:04.262449 9810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:04.262531 9805 server_base.cc:1034] running on GCE node
W20260928 14:36:04.262454 9813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:04.263331 9805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:04.264465 9805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:04.265666 9805 hybrid_clock.cc:648] HybridClock initialized: now 1790606164265650 us; error 27 us; skew 500 ppm
Sep 28 14:36:04 dist-test-slave-06lh krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606164, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:04.268498 9805 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:36:04.269394 9805 webserver.cc:460] Webserver started at http://127.6.23.254:43299/ using document root <none> and password file <none>
I20260928 14:36:04.269984 9805 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:04.270076 9805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:04.270314 9805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:04.272580 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "1cf29abfceee4742b535df08237ac1c0"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "c4e24fca2068dea4435bf921d306e62d"
server_key_iv: "f7fa27dc7f3dfb33389010d6cc87d45b"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.273211 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "1cf29abfceee4742b535df08237ac1c0"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "c4e24fca2068dea4435bf921d306e62d"
server_key_iv: "f7fa27dc7f3dfb33389010d6cc87d45b"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.277248 9805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:04.279807 9820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:04.281179 9805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:04.281352 9805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "1cf29abfceee4742b535df08237ac1c0"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "c4e24fca2068dea4435bf921d306e62d"
server_key_iv: "f7fa27dc7f3dfb33389010d6cc87d45b"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.281490 9805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:04.325136 9805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:04.328464 9805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:04.328706 9805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:04.336210 9805 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:40495
I20260928 14:36:04.336231 9872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:40495 every 8 connection(s)
I20260928 14:36:04.337242 9805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:04.340167 9873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:04.344499 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 9805
I20260928 14:36:04.344636 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:04.344915 6239 external_mini_cluster.cc:1428] Setting key eec865e00a42f48e6971d30bf92ccc07
I20260928 14:36:04.346549 9873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0: Bootstrap starting.
I20260928 14:36:04.348933 9873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:04.349700 9873 log.cc:824] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:04.351749 9873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0: No bootstrap required, opened a new log
Sep 28 14:36:04 dist-test-slave-06lh krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:04.356225 9873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf29abfceee4742b535df08237ac1c0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40495 } }
I20260928 14:36:04.356508 9873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:04.356593 9873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf29abfceee4742b535df08237ac1c0, State: Initialized, Role: FOLLOWER
I20260928 14:36:04.357045 9873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf29abfceee4742b535df08237ac1c0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40495 } }
I20260928 14:36:04.357163 9873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:04.357232 9873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:04.357331 9873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:04.358551 9873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf29abfceee4742b535df08237ac1c0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40495 } }
I20260928 14:36:04.358952 9873 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf29abfceee4742b535df08237ac1c0; no voters:
I20260928 14:36:04.359373 9873 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:04.359561 9878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:04.359787 9878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf29abfceee4742b535df08237ac1c0, State: Running, Role: LEADER
I20260928 14:36:04.360105 9878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf29abfceee4742b535df08237ac1c0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40495 } }
I20260928 14:36:04.360752 9873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:04.360850 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:04.346453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.6.23.254:40495)
0928 14:36:04.346943 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:04.346959 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:04.346999 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:36:04.347765 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:04.347785 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:04.348179 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:04.348530 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:04.349390 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.350668 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.351625 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.352191 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.352374 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:04.355292 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:04.355298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:04.355313 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:04.355342 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:04.357324 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:04.357847 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:04.357851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:04.357856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:04.357991 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:04.358285 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:04.358288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:04.358289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:04.358765 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:04.358923 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:04.359349 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 14:36:04.359697 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":145,"threads_started":1}
I20260928 14:36:04.362732 9879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cf29abfceee4742b535df08237ac1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf29abfceee4742b535df08237ac1c0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40495 } } }
I20260928 14:36:04.362732 9880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf29abfceee4742b535df08237ac1c0. Latest consensus state: current_term: 1 leader_uuid: "1cf29abfceee4742b535df08237ac1c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf29abfceee4742b535df08237ac1c0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40495 } } }
I20260928 14:36:04.362888 9880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:04.362888 9879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:04.363318 9893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:04.366309 9893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:04.372612 9893 catalog_manager.cc:1347] Generated new cluster ID: a51a5c40b2ff4131ae23594dda79d0d6
I20260928 14:36:04.372689 9893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:04.399179 9893 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:04.400557 9893 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:04.416339 9893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cf29abfceee4742b535df08237ac1c0: Generated new TSK 0
I20260928 14:36:04.417284 9893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:36:04.446369 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40495
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:04.561513 9899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:04.561890 9899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:04.562003 9899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:04.566390 9899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:04.566510 9899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:04.566617 9899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:04.572340 9899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:04.573514 9899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:04.574604 9899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:04.585023 9905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:04.585034 9904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:04.585132 9899 server_base.cc:1034] running on GCE node
W20260928 14:36:04.585067 9907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:04.585587 9899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:04.586364 9899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:04.587555 9899 hybrid_clock.cc:648] HybridClock initialized: now 1790606164587524 us; error 41 us; skew 500 ppm
Sep 28 14:36:04 dist-test-slave-06lh krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606164, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:04.590306 9899 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:36:04.591355 9899 webserver.cc:460] Webserver started at http://127.6.23.193:33643/ using document root <none> and password file <none>
I20260928 14:36:04.591943 9899 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:04.592034 9899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:04.592252 9899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:04.594321 9899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "a7f0c0ae49ae4218823c922ee50f28bd"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "eca535847812a892a2f036ac91006264"
server_key_iv: "40f4d896ae8b96ff80c28f5e1cb4b787"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.594821 9899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "a7f0c0ae49ae4218823c922ee50f28bd"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "eca535847812a892a2f036ac91006264"
server_key_iv: "40f4d896ae8b96ff80c28f5e1cb4b787"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.598997 9899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 14:36:04.602141 9914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:04.603580 9899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:04.603717 9899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "a7f0c0ae49ae4218823c922ee50f28bd"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "eca535847812a892a2f036ac91006264"
server_key_iv: "40f4d896ae8b96ff80c28f5e1cb4b787"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.603837 9899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:04.613947 9899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:04.617018 9899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:04.617220 9899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:04.617803 9899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:04.618619 9899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:04.618690 9899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:04.618762 9899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:04.618815 9899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:04.627614 9899 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:41243
I20260928 14:36:04.627648 10027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:41243 every 8 connection(s)
I20260928 14:36:04.628552 9899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:04.633399 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 9899
I20260928 14:36:04.633539 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:04.633821 6239 external_mini_cluster.cc:1428] Setting key c68f1fae523882b888da1c86bb2a484e
Sep 28 14:36:04 dist-test-slave-06lh krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606164, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:36:04.643420 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:04.630490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:57897 (local address 127.6.23.254:40495)
0928 14:36:04.630644 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:04.630648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:04.631454 (+ 806us) server_negotiation.cc:407] Connection header received
0928 14:36:04.632852 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:04.632856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:04.632925 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:04.633032 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:04.634391 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.635158 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.636350 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.636597 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.636719 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:04.639582 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:04.639586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:04.639589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:04.639623 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:04.641178 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:04.641725 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:04.641729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:04.641731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:04.641817 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:04.642165 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:04.642168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:04.642169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:04.642386 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:04.642498 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:04.643054 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 14:36:04.643192 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:36:04.644292 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:04.630814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40495 (local address 127.6.23.193:57897)
0928 14:36:04.631312 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:04.631355 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:04.632637 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:04.633213 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:04.633220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:04.633678 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:04.634221 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:04.634237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.635314 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.635317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:04.636215 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:04.636225 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.636759 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.636764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:04.636949 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:04.637587 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:04.637604 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:04.639397 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:04.641317 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:04.641324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:04.641343 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:04.641612 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:04.641912 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:04.641916 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:04.641919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:04.642061 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:04.642496 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:04.642504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:04.642795 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 14:36:04.642999 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 14:36:04.643382 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:36:04.645684 10028 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40495
I20260928 14:36:04.645977 10028 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:04.646574 10028 heartbeater.cc:511] Master 127.6.23.254:40495 requested a full tablet report, sending...
I20260928 14:36:04.648420 9837 ts_manager.cc:194] Registered new tserver with Master: a7f0c0ae49ae4218823c922ee50f28bd (127.6.23.193:41243)
I20260928 14:36:04.650143 9837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:57897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:36:04.663959 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40495
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:04.778450 10033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:04.778746 10033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:04.778791 10033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:04.782996 10033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:04.783058 10033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:04.783118 10033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:04.788532 10033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:04.789546 10033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:04.790510 10033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:36:04.800420 10033 server_base.cc:1034] running on GCE node
W20260928 14:36:04.800482 10038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:04.800499 10039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:04.800408 10041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:04.801134 10033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:04.801749 10033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:04.802917 10033 hybrid_clock.cc:648] HybridClock initialized: now 1790606164802903 us; error 27 us; skew 500 ppm
Sep 28 14:36:04 dist-test-slave-06lh krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606164, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:04.805876 10033 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:36:04.806910 10033 webserver.cc:460] Webserver started at http://127.6.23.194:39721/ using document root <none> and password file <none>
I20260928 14:36:04.807583 10033 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:04.807678 10033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:04.807924 10033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:04.810052 10033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "b5d84db93a4943d6bdd0c304d0aeb2c3"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "442faf4b529b3d90019ccc606c247e4b"
server_key_iv: "43c884cd39fdf248a18ad687cfa96517"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.810642 10033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "b5d84db93a4943d6bdd0c304d0aeb2c3"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "442faf4b529b3d90019ccc606c247e4b"
server_key_iv: "43c884cd39fdf248a18ad687cfa96517"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.814520 10033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:36:04.817054 10048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:04.818236 10033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:04.818375 10033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "b5d84db93a4943d6bdd0c304d0aeb2c3"
format_stamp: "Formatted at 2026-09-28 14:36:04 on dist-test-slave-06lh"
server_key: "442faf4b529b3d90019ccc606c247e4b"
server_key_iv: "43c884cd39fdf248a18ad687cfa96517"
server_key_version: "encryptionkey@0"
I20260928 14:36:04.818488 10033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:04.845947 10033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:04.849324 10033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:04.849555 10033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:04.850206 10033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:04.851130 10033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:04.851203 10033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:04.851292 10033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:04.851336 10033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:04.860512 10033 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:41021
I20260928 14:36:04.860533 10161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:41021 every 8 connection(s)
I20260928 14:36:04.861636 10033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:04.871124 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 10033
I20260928 14:36:04.871272 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:04.871619 6239 external_mini_cluster.cc:1428] Setting key 6e05856178b117ba2bb6e64a460e5461
Sep 28 14:36:04 dist-test-slave-06lh krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606164, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:04.876878 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:04.863887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:56379 (local address 127.6.23.254:40495)
0928 14:36:04.864040 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:04.864045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:04.864974 (+ 929us) server_negotiation.cc:407] Connection header received
0928 14:36:04.866522 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:04.866525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:04.866584 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:04.866669 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:04.868020 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.868576 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.869769 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.870008 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.870124 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:04.872956 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:04.872960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:04.872962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:04.872997 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:04.874493 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:04.875083 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:04.875088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:04.875092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:04.875193 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:04.875558 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:04.875563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:04.875566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:04.875846 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:04.875967 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:04.876483 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 14:36:04.876631 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:36:04.877745 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:04.864316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40495 (local address 127.6.23.194:56379)
0928 14:36:04.864804 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:04.864864 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:04.866283 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:04.866873 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:04.866880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:04.867350 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:04.867866 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:04.867881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.868715 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.868724 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:04.869638 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:04.869647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:04.870167 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:04.870172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:04.870346 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:04.870973 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:04.870995 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:04.872777 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:04.874620 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:04.874634 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:04.874659 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:04.874978 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:04.875312 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:04.875315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:04.875316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:04.875464 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:04.875970 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:04.875975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:04.876220 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 14:36:04.876396 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 14:36:04.876677 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260928 14:36:04.879139 10162 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40495
I20260928 14:36:04.879479 10162 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:04.880125 10162 heartbeater.cc:511] Master 127.6.23.254:40495 requested a full tablet report, sending...
I20260928 14:36:04.881299 9837 ts_manager.cc:194] Registered new tserver with Master: b5d84db93a4943d6bdd0c304d0aeb2c3 (127.6.23.194:41021)
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:36:04.881970 9837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:56379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:36:04.900162 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40495
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:05.016640 10167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:05.016929 10167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:05.016974 10167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:05.021104 10167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:05.021188 10167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:05.021250 10167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:05.026800 10167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:05.027980 10167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:05.029000 10167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:05.038851 10175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:05.038870 10173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:05.039122 10167 server_base.cc:1034] running on GCE node
W20260928 14:36:05.039284 10172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:05.039677 10167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:05.040334 10167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:05.041503 10167 hybrid_clock.cc:648] HybridClock initialized: now 1790606165041486 us; error 31 us; skew 500 ppm
Sep 28 14:36:05 dist-test-slave-06lh krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606165, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:05.044512 10167 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:36:05.045450 10167 webserver.cc:460] Webserver started at http://127.6.23.195:44099/ using document root <none> and password file <none>
I20260928 14:36:05.046104 10167 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:05.046170 10167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:05.046422 10167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:05.048635 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "5309a02a86f6434c8d1c9026beae4a7a"
format_stamp: "Formatted at 2026-09-28 14:36:05 on dist-test-slave-06lh"
server_key: "18ffea347dbcfb767ad94a69127d60c0"
server_key_iv: "4817308c060b9057005c428945e7a157"
server_key_version: "encryptionkey@0"
I20260928 14:36:05.049245 10167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "5309a02a86f6434c8d1c9026beae4a7a"
format_stamp: "Formatted at 2026-09-28 14:36:05 on dist-test-slave-06lh"
server_key: "18ffea347dbcfb767ad94a69127d60c0"
server_key_iv: "4817308c060b9057005c428945e7a157"
server_key_version: "encryptionkey@0"
I20260928 14:36:05.053184 10167 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:05.055771 10182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:05.057034 10167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:05.057160 10167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "5309a02a86f6434c8d1c9026beae4a7a"
format_stamp: "Formatted at 2026-09-28 14:36:05 on dist-test-slave-06lh"
server_key: "18ffea347dbcfb767ad94a69127d60c0"
server_key_iv: "4817308c060b9057005c428945e7a157"
server_key_version: "encryptionkey@0"
I20260928 14:36:05.057271 10167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:05.069620 10167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:05.073248 10167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:05.073463 10167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:05.074090 10167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:05.074942 10167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:05.074990 10167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:05.075062 10167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:05.075098 10167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:05.084367 10167 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:46689
I20260928 14:36:05.084386 10295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:46689 every 8 connection(s)
I20260928 14:36:05.085307 10167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:05.086654 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 10167
I20260928 14:36:05.086792 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:05.087083 6239 external_mini_cluster.cc:1428] Setting key 32d5c01e5796d15c50f3604338574aea
Sep 28 14:36:05 dist-test-slave-06lh krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606165, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:05.099805 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:05.087444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:57569 (local address 127.6.23.254:40495)
0928 14:36:05.087611 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:05.087615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:05.088783 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 14:36:05.090320 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:05.090323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:05.090390 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:05.090484 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:05.091931 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:05.092476 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:05.093626 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:05.093853 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:05.093962 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:05.096380 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:05.096384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:05.096386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:05.096410 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:05.097654 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:05.098165 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:05.098168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:05.098169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:05.098246 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:05.098591 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:05.098601 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:05.098603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:05.098781 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:05.098901 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:05.099472 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 14:36:05.099599 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:36:05.100903 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:05.088065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40495 (local address 127.6.23.195:57569)
0928 14:36:05.088647 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:05.088689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:05.090094 (+ 1405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:05.090673 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:05.090680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:05.091092 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:05.091794 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:05.091811 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:05.092602 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:05.092606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:05.093480 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:05.093488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:05.094001 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:05.094005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:05.094176 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:05.094694 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:05.094709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:05.096206 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:05.097779 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:05.097784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:05.097800 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:05.098051 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:05.098336 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:05.098341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:05.098343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:05.098480 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:05.098898 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:05.098905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:05.099157 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 14:36:05.099393 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 14:36:05.099742 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":148,"threads_started":1}
I20260928 14:36:05.101999 10296 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40495
I20260928 14:36:05.102262 10296 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:05.102813 10296 heartbeater.cc:511] Master 127.6.23.254:40495 requested a full tablet report, sending...
I20260928 14:36:05.103900 9837 ts_manager.cc:194] Registered new tserver with Master: 5309a02a86f6434c8d1c9026beae4a7a (127.6.23.195:46689)
I20260928 14:36:05.104440 9837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:57569
I20260928 14:36:05.113101 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:36:05 dist-test-slave-06lh krb5kdc[9792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:36:05 dist-test-slave-06lh krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:36:05.139689 10303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:05.129818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.6.23.193:41243)
0928 14:36:05.130147 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:05.130169 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:05.130196 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:36:05.130356 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:05.130364 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:05.130582 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:05.130713 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:05.131497 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:05.132309 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:05.133127 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:05.133527 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:05.133615 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:05.136062 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:05.136072 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:05.136087 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:05.136116 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:05.137942 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:05.138336 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:05.138341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:05.138342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:05.138425 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:05.138652 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:05.138654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:05.138655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:05.138918 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:05.139093 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:05.139293 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 14:36:05.139430 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":110,"threads_started":1}
Sep 28 14:36:05 dist-test-slave-06lh krb5kdc[9792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:05.150861 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:05.142846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54098 (local address 127.6.23.254:40495)
0928 14:36:05.142998 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:05.143002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:05.143016 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:36:05.143061 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:05.143064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:05.143104 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:05.143174 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:05.143862 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:05.144446 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:05.145242 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:05.145500 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:05.145592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:05.148037 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:05.148040 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:05.148042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:05.148069 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:05.149364 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:05.149755 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:05.149757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:05.149759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:05.149835 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:05.150055 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:05.150057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:05.150058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:05.150278 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:05.150389 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:05.150626 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 14:36:05.150735 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 14:36:05.151362 9837 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54098
I20260928 14:36:05.152112 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 9899
I20260928 14:36:05.157867 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 10033
I20260928 14:36:05.163697 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 10167
I20260928 14:36:05.169253 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 9805
2026-09-28T14:36:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2199 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:05 dist-test-slave-06lh krb5kdc[10308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:05 dist-test-slave-06lh krb5kdc[10308](info): set up 2 sockets
Sep 28 14:36:05 dist-test-slave-06lh krb5kdc[10308](info): commencing operation
krb5kdc: starting...
W20260928 14:36:06.211967 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:06 dist-test-slave-06lh krb5kdc[10308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:36:06.338654 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39827
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:39827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:06.454147 10321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:06.454463 10321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:06.454507 10321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:06.459144 10321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:06.459244 10321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:06.459266 10321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:06.459285 10321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:06.459300 10321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:06.464854 10321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:39827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:06.465942 10321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:06.466984 10321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:06.476630 10329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:06.476644 10326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:06.476675 10327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:06.476924 10321 server_base.cc:1034] running on GCE node
I20260928 14:36:06.477697 10321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:06.478938 10321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:06.480101 10321 hybrid_clock.cc:648] HybridClock initialized: now 1790606166480056 us; error 57 us; skew 500 ppm
Sep 28 14:36:06 dist-test-slave-06lh krb5kdc[10308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606166, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:06.489691 10321 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:36:06.490648 10321 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:06.490720 10321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:06.490893 10321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:06.493562 10321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "faf8dee7dc724f67809afd06039d9dc4"
format_stamp: "Formatted at 2026-09-28 14:36:06 on dist-test-slave-06lh"
server_key: "b82cc3e5fe72a7a0fcb05b751fb255b8"
server_key_iv: "80dbd6dfa523a9401705bbd4b3b21669"
server_key_version: "encryptionkey@0"
I20260928 14:36:06.494199 10321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "faf8dee7dc724f67809afd06039d9dc4"
format_stamp: "Formatted at 2026-09-28 14:36:06 on dist-test-slave-06lh"
server_key: "b82cc3e5fe72a7a0fcb05b751fb255b8"
server_key_iv: "80dbd6dfa523a9401705bbd4b3b21669"
server_key_version: "encryptionkey@0"
I20260928 14:36:06.498564 10321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:36:06.502060 10335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:06.503652 10321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:06.503839 10321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "faf8dee7dc724f67809afd06039d9dc4"
format_stamp: "Formatted at 2026-09-28 14:36:06 on dist-test-slave-06lh"
server_key: "b82cc3e5fe72a7a0fcb05b751fb255b8"
server_key_iv: "80dbd6dfa523a9401705bbd4b3b21669"
server_key_version: "encryptionkey@0"
I20260928 14:36:06.503978 10321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:06.530673 10321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:06.534246 10321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:06.534463 10321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:06.542721 10321 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:39827
I20260928 14:36:06.543015 10387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:39827 every 8 connection(s)
I20260928 14:36:06.543532 10321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:06.545536 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 10321
I20260928 14:36:06.545681 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:06.545995 6239 external_mini_cluster.cc:1428] Setting key 9206e9cfd4588d8ad69a715f35987f92
I20260928 14:36:06.546945 10388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:06.554217 10388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4: Bootstrap starting.
Sep 28 14:36:06 dist-test-slave-06lh krb5kdc[10308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:06.556931 10388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:06.557711 10388 log.cc:824] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:06.559854 10388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4: No bootstrap required, opened a new log
I20260928 14:36:06.561691 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:06.547547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42358 (local address 127.6.23.254:39827)
0928 14:36:06.547992 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:06.548009 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:06.548048 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:36:06.548788 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:06.548807 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:06.549158 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:06.549456 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:06.550426 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:06.551568 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:06.552819 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:06.553424 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:06.553553 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:06.556391 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:06.556396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:06.556412 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:06.556442 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:06.558646 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:06.559101 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:06.559107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:06.559113 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:06.559285 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:06.559591 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:06.559595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:06.559597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:06.560062 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:06.560279 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:06.560547 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 14:36:06.560865 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":145,"threads_started":1}
I20260928 14:36:06.563947 10388 raft_consensus.cc:348] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8dee7dc724f67809afd06039d9dc4" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39827 } }
I20260928 14:36:06.564275 10388 raft_consensus.cc:374] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:06.564344 10388 raft_consensus.cc:729] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf8dee7dc724f67809afd06039d9dc4, State: Initialized, Role: FOLLOWER
I20260928 14:36:06.564765 10388 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8dee7dc724f67809afd06039d9dc4" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39827 } }
I20260928 14:36:06.564888 10388 raft_consensus.cc:388] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:06.564934 10388 raft_consensus.cc:482] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:06.565007 10388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:06.588721 10388 raft_consensus.cc:504] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8dee7dc724f67809afd06039d9dc4" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39827 } }
I20260928 14:36:06.589303 10388 leader_election.cc:302] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf8dee7dc724f67809afd06039d9dc4; no voters:
I20260928 14:36:06.589797 10388 leader_election.cc:288] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:06.589941 10393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:06.590221 10393 raft_consensus.cc:686] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [term 1 LEADER]: Becoming Leader. State: Replica: faf8dee7dc724f67809afd06039d9dc4, State: Running, Role: LEADER
I20260928 14:36:06.590680 10393 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8dee7dc724f67809afd06039d9dc4" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39827 } }
I20260928 14:36:06.591446 10388 sys_catalog.cc:564] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:06.592711 10394 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faf8dee7dc724f67809afd06039d9dc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8dee7dc724f67809afd06039d9dc4" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39827 } } }
I20260928 14:36:06.592851 10394 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:06.593186 10395 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf8dee7dc724f67809afd06039d9dc4. Latest consensus state: current_term: 1 leader_uuid: "faf8dee7dc724f67809afd06039d9dc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf8dee7dc724f67809afd06039d9dc4" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39827 } } }
I20260928 14:36:06.593323 10395 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:06.593437 10402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:06.596918 10402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:06.602885 10402 catalog_manager.cc:1347] Generated new cluster ID: 889edf17d5bc43dcba0fbca7bfedb1a5
I20260928 14:36:06.602979 10402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:06.612416 10402 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:06.613678 10402 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:06.626500 10402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4: Generated new TSK 0
I20260928 14:36:06.627466 10402 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
W20260928 14:36:07.585363 6239 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.23.193: real 0.952s user 0.000s sys 0.004s
I20260928 14:36:07.585883 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39827
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:07.709941 10414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:07.710527 10414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:07.710623 10414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:07.717535 10414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:07.717615 10414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:07.717720 10414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:07.725859 10414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:07.727409 10414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:07.728883 10414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:07.741130 10422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:07.741187 10419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:07.741179 10420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:07.742026 10414 server_base.cc:1034] running on GCE node
I20260928 14:36:07.742616 10414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:07.743554 10414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:07.744755 10414 hybrid_clock.cc:648] HybridClock initialized: now 1790606167744726 us; error 67 us; skew 500 ppm
Sep 28 14:36:07 dist-test-slave-06lh krb5kdc[10308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606167, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:07.748063 10414 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:36:07.749007 10414 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:07.749091 10414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:07.749311 10414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:07.751456 10414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "0b6ec0de108f439f88f27c166aeabb62"
format_stamp: "Formatted at 2026-09-28 14:36:07 on dist-test-slave-06lh"
server_key: "7b6ba8fc76e349d0a7f99d7b2fc590f9"
server_key_iv: "b6e176b71acfe86deee07c3ff10366c6"
server_key_version: "encryptionkey@0"
I20260928 14:36:07.751976 10414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "0b6ec0de108f439f88f27c166aeabb62"
format_stamp: "Formatted at 2026-09-28 14:36:07 on dist-test-slave-06lh"
server_key: "7b6ba8fc76e349d0a7f99d7b2fc590f9"
server_key_iv: "b6e176b71acfe86deee07c3ff10366c6"
server_key_version: "encryptionkey@0"
I20260928 14:36:07.755746 10414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:36:07.758100 10428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:07.759414 10414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:07.759506 10414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "0b6ec0de108f439f88f27c166aeabb62"
format_stamp: "Formatted at 2026-09-28 14:36:07 on dist-test-slave-06lh"
server_key: "7b6ba8fc76e349d0a7f99d7b2fc590f9"
server_key_iv: "b6e176b71acfe86deee07c3ff10366c6"
server_key_version: "encryptionkey@0"
I20260928 14:36:07.759590 10414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:07.781680 10414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:07.784794 10414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:07.784951 10414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:07.785492 10414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:07.786260 10414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:07.786306 10414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:07.786345 10414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:07.786361 10414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:07.795775 10414 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:38125
I20260928 14:36:07.795794 10541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:38125 every 8 connection(s)
I20260928 14:36:07.796437 10414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:07.802924 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 10414
I20260928 14:36:07.803056 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:07.803387 6239 external_mini_cluster.cc:1428] Setting key 514182d65cc963fa8dd3b75105efbad3
Sep 28 14:36:07 dist-test-slave-06lh krb5kdc[10308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606167, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:07.811522 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:07.798208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:58399 (local address 127.6.23.254:39827)
0928 14:36:07.798478 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:07.798481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:07.799652 (+ 1171us) server_negotiation.cc:407] Connection header received
0928 14:36:07.800540 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:07.800544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:07.800605 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:07.800725 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:07.801897 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:07.802440 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:07.804038 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:07.804274 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:07.804390 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:07.807224 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:07.807249 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:07.807254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:07.807290 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:07.809113 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:07.809669 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:07.809676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:07.809682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:07.809785 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:07.810129 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:07.810135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:07.810138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:07.810415 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:07.810537 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:07.811068 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 14:36:07.811257 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":145,"threads_started":1}
I20260928 14:36:07.812273 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:07.798604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39827 (local address 127.6.23.193:58399)
0928 14:36:07.799506 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:07.799550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:07.800330 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:07.800905 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:07.800912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:07.801261 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:07.801741 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:07.801755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:07.802565 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:07.802571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:07.803912 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:07.803921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:07.804435 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:07.804439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:07.804599 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:07.805197 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:07.805217 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:07.807016 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:07.809253 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:07.809259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:07.809272 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:07.809578 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:07.809893 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:07.809895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:07.809897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:07.810036 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:07.810554 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:07.810560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:07.810811 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 14:36:07.810980 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 14:36:07.811323 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260928 14:36:07.813619 10542 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39827
I20260928 14:36:07.813860 10542 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:36:07.814338 10542 heartbeater.cc:511] Master 127.6.23.254:39827 requested a full tablet report, sending...
I20260928 14:36:07.815871 10352 ts_manager.cc:194] Registered new tserver with Master: 0b6ec0de108f439f88f27c166aeabb62 (127.6.23.193:38125)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:36:07.817420 10352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:58399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:36:07.834092 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39827
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:07.948544 10548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:07.948825 10548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:07.948904 10548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:07.953096 10548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:07.953161 10548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:07.953264 10548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:07.958673 10548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:07.959867 10548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:07.960920 10548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:07.970654 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:07.970649 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:07.970813 10548 server_base.cc:1034] running on GCE node
W20260928 14:36:07.970731 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:07.971427 10548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:07.972025 10548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:07.973186 10548 hybrid_clock.cc:648] HybridClock initialized: now 1790606167973171 us; error 28 us; skew 500 ppm
Sep 28 14:36:07 dist-test-slave-06lh krb5kdc[10308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606167, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:07.976073 10548 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:36:07.976861 10548 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:07.976924 10548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:07.977152 10548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:07.979301 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8"
format_stamp: "Formatted at 2026-09-28 14:36:07 on dist-test-slave-06lh"
server_key: "b335c91dfb4ae0e46a39a23edc68e13c"
server_key_iv: "d32bd54954bb6561ec508f4e195f80e2"
server_key_version: "encryptionkey@0"
I20260928 14:36:07.979925 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8"
format_stamp: "Formatted at 2026-09-28 14:36:07 on dist-test-slave-06lh"
server_key: "b335c91dfb4ae0e46a39a23edc68e13c"
server_key_iv: "d32bd54954bb6561ec508f4e195f80e2"
server_key_version: "encryptionkey@0"
I20260928 14:36:07.983652 10548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:36:07.986145 10562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:07.987442 10548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:07.987583 10548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8"
format_stamp: "Formatted at 2026-09-28 14:36:07 on dist-test-slave-06lh"
server_key: "b335c91dfb4ae0e46a39a23edc68e13c"
server_key_iv: "d32bd54954bb6561ec508f4e195f80e2"
server_key_version: "encryptionkey@0"
I20260928 14:36:07.987924 10548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:08.002811 10548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:08.005777 10548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:08.005990 10548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:08.006582 10548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:08.007486 10548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:08.007560 10548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:08.007633 10548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:08.007675 10548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:08.017107 10548 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:42781
I20260928 14:36:08.017122 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:42781 every 8 connection(s)
I20260928 14:36:08.017742 10548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:08.020581 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 10548
I20260928 14:36:08.020716 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:08.021037 6239 external_mini_cluster.cc:1428] Setting key 991fe337d160cace40138814f642cb16
Sep 28 14:36:08 dist-test-slave-06lh krb5kdc[10308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606167, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:36:08.032759 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.019698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:37739 (local address 127.6.23.254:39827)
0928 14:36:08.019864 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.019868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.020669 (+ 801us) server_negotiation.cc:407] Connection header received
0928 14:36:08.022142 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.022145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.022195 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.022276 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:08.023578 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.024251 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.025467 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.025908 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.026071 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.028818 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:08.028821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:08.028823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:08.028848 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:08.030382 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:08.030974 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:08.030978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:08.030990 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:08.031066 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:08.031504 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:08.031507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:08.031509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:08.031701 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:08.031808 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.032375 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.032570 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:36:08.033620 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.020045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39827 (local address 127.6.23.194:37739)
0928 14:36:08.020504 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.020550 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:08.021922 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:08.022454 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:08.022462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:08.022831 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:08.023421 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.023438 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.024417 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.024420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.025342 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.025353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.026084 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.026086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.026238 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.026954 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:08.026971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:08.028626 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:08.030547 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:08.030553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:08.030572 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:08.030861 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:08.031160 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:08.031164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:08.031166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:08.031380 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:08.031818 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:08.031827 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:08.032095 (+ 268us) client_negotiation.cc:769] Sending connection context
0928 14:36:08.032290 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 14:36:08.032660 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
I20260928 14:36:08.034909 10676 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39827
I20260928 14:36:08.035157 10676 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:08.035722 10676 heartbeater.cc:511] Master 127.6.23.254:39827 requested a full tablet report, sending...
I20260928 14:36:08.036871 10352 ts_manager.cc:194] Registered new tserver with Master: bdca84f91d1f4e9ba0cde60d4ea07dd8 (127.6.23.194:42781)
I20260928 14:36:08.037763 10352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:37739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:36:08.051039 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39827
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:08.162009 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:08.162314 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:08.162390 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:08.166501 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:08.166587 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:08.166695 10681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:08.172019 10681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:08.173090 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:08.174115 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:08.183553 10686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:08.183599 10687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:08.183564 10689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:08.183758 10681 server_base.cc:1034] running on GCE node
I20260928 14:36:08.184185 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:08.184847 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:08.186030 10681 hybrid_clock.cc:648] HybridClock initialized: now 1790606168186008 us; error 30 us; skew 500 ppm
Sep 28 14:36:08 dist-test-slave-06lh krb5kdc[10308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606168, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:08.188833 10681 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:36:08.189615 10681 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:08.189709 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:08.189944 10681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:08.192238 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c"
format_stamp: "Formatted at 2026-09-28 14:36:08 on dist-test-slave-06lh"
server_key: "07eb7cf3990cd97b0b6692635cb6dae2"
server_key_iv: "56a236a90773dd1746dd7045bba2b25c"
server_key_version: "encryptionkey@0"
I20260928 14:36:08.192900 10681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c"
format_stamp: "Formatted at 2026-09-28 14:36:08 on dist-test-slave-06lh"
server_key: "07eb7cf3990cd97b0b6692635cb6dae2"
server_key_iv: "56a236a90773dd1746dd7045bba2b25c"
server_key_version: "encryptionkey@0"
I20260928 14:36:08.196843 10681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 14:36:08.199401 10695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:08.200680 10681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:08.200819 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c"
format_stamp: "Formatted at 2026-09-28 14:36:08 on dist-test-slave-06lh"
server_key: "07eb7cf3990cd97b0b6692635cb6dae2"
server_key_iv: "56a236a90773dd1746dd7045bba2b25c"
server_key_version: "encryptionkey@0"
I20260928 14:36:08.200937 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:08.219388 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:08.222488 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:08.222712 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:08.223333 10681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:08.224184 10681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:08.224255 10681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:08.224329 10681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:08.224373 10681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:08.233261 10681 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:42587
I20260928 14:36:08.233279 10808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:42587 every 8 connection(s)
I20260928 14:36:08.233868 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:08.237615 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 10681
I20260928 14:36:08.237739 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:08.238056 6239 external_mini_cluster.cc:1428] Setting key 2dc156d9b326f351214cb849769cf0c8
Sep 28 14:36:08 dist-test-slave-06lh krb5kdc[10308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606168, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:08.247666 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.235677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:38057 (local address 127.6.23.254:39827)
0928 14:36:08.235813 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.235817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.236944 (+ 1127us) server_negotiation.cc:407] Connection header received
0928 14:36:08.238092 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.238094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.238174 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.238255 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:08.239904 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.240589 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.241701 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.241914 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.241997 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.244420 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:08.244423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:08.244425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:08.244448 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:08.245673 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:08.246130 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:08.246132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:08.246134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:08.246204 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:08.246482 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:08.246484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:08.246485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:08.246650 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:08.246724 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.247427 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.247543 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:36:08.248471 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.236006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39827 (local address 127.6.23.195:38057)
0928 14:36:08.236783 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.236824 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:08.237827 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:08.238475 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:08.238483 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:08.239086 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:08.239767 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.239782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.240705 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.240708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.241597 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.241606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.242031 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.242034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.242167 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.242779 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:08.242795 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:08.244270 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:08.245772 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:08.245777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:08.245790 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:08.246038 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:08.246278 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:08.246280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:08.246281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:08.246406 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:08.246764 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:08.246770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:08.247046 (+ 276us) client_negotiation.cc:769] Sending connection context
0928 14:36:08.247280 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 14:36:08.247591 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
I20260928 14:36:08.249683 10809 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39827
I20260928 14:36:08.249946 10809 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:08.250460 10809 heartbeater.cc:511] Master 127.6.23.254:39827 requested a full tablet report, sending...
I20260928 14:36:08.251619 10352 ts_manager.cc:194] Registered new tserver with Master: ff22e4422cc44b4cb6aae5ebc5ca5a3c (127.6.23.195:42587)
I20260928 14:36:08.252228 10352 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:38057
I20260928 14:36:08.252245 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:36:08.261770 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.254461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.6.23.254:39827)
0928 14:36:08.254621 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.254626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.254814 (+ 188us) server_negotiation.cc:407] Connection header received
0928 14:36:08.254929 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.254931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.254969 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.255033 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:08.255760 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.256279 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.257129 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.257338 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.257426 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.258811 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:08.258814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:08.258816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:08.258839 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:08.260252 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:08.260645 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:08.260648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:08.260649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:08.260718 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:08.260934 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:08.260936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:08.260937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:08.261147 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:08.261246 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.261461 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.261555 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:36:08.264746 10352 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:36:08.267719 10352 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:08.282305 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.276123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:42781 (local address 127.0.0.1:35308)
0928 14:36:08.276968 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.276982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:08.277409 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:08.278155 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:08.278158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:08.278179 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:08.278290 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.278295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.279365 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.279368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.280765 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.280775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.281824 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.281828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.281953 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.282000 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:36:08.282095 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:36:08.282184 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":51,"spinlock_wait_cycles":775808,"thread_start_us":103,"threads_started":1}
I20260928 14:36:08.282392 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.277255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.6.23.194:42781)
0928 14:36:08.277676 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.277689 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.277709 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:36:08.277783 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.277787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.277977 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.278149 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:08.278398 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.279178 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.280889 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.281683 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.281908 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.282060 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.282123 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.282218 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":113,"threads_started":1}
I20260928 14:36:08.282593 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.276034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38125 (local address 127.0.0.1:56992)
0928 14:36:08.276794 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.276850 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:08.277262 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:08.277936 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:08.277941 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:08.277962 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:08.278133 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.278143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.279672 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.279676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.280982 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.280992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.282173 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.282176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.282260 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.282284 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:36:08.282366 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:36:08.282449 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":39,"thread_start_us":96,"threads_started":1}
I20260928 14:36:08.282701 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.276158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.6.23.193:38125)
0928 14:36:08.276508 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.276515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.276957 (+ 442us) server_negotiation.cc:407] Connection header received
0928 14:36:08.277599 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.277604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.277779 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.277935 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:08.278268 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.279523 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.281127 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.281895 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.282001 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.282114 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.282423 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.282538 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":63,"threads_started":1}
I20260928 14:36:08.282964 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.276750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:42587 (local address 127.0.0.1:43324)
0928 14:36:08.277305 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.277319 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:08.277416 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:08.278284 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:08.278287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:08.278305 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:08.278420 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.278426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.279781 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.279784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.281001 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.281007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.281790 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.281794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.282682 (+ 888us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.282711 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:36:08.282783 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:36:08.282857 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":60,"spinlock_wait_cycles":4480,"thread_start_us":93,"threads_started":1}
I20260928 14:36:08.285360 10610 tablet_service.cc:1467] Processing CreateTablet for tablet 26e41eb9d3a742d590bdb6451bc49ee7 (DEFAULT_TABLE table=test-table [id=55ec4abe0d0442b4bd3a7171e3bc08d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:08.285544 10476 tablet_service.cc:1467] Processing CreateTablet for tablet 26e41eb9d3a742d590bdb6451bc49ee7 (DEFAULT_TABLE table=test-table [id=55ec4abe0d0442b4bd3a7171e3bc08d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:08.286099 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.276925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.6.23.195:42587)
0928 14:36:08.277677 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.277689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.277709 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:36:08.277783 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.277787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.277977 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.278125 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:08.278548 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.279656 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.281128 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.281666 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.281791 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.281944 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.283196 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.285940 (+ 2744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":100,"threads_started":1}
I20260928 14:36:08.286464 10610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e41eb9d3a742d590bdb6451bc49ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:08.286523 10476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e41eb9d3a742d590bdb6451bc49ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:08.288894 10743 tablet_service.cc:1467] Processing CreateTablet for tablet 26e41eb9d3a742d590bdb6451bc49ee7 (DEFAULT_TABLE table=test-table [id=55ec4abe0d0442b4bd3a7171e3bc08d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:08.289976 10743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26e41eb9d3a742d590bdb6451bc49ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:08.293475 10827 tablet_bootstrap.cc:492] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62: Bootstrap starting.
I20260928 14:36:08.294853 10828 tablet_bootstrap.cc:492] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8: Bootstrap starting.
I20260928 14:36:08.296018 10827 tablet_bootstrap.cc:654] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:08.297044 10828 tablet_bootstrap.cc:654] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:08.297036 10829 tablet_bootstrap.cc:492] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Bootstrap starting.
I20260928 14:36:08.297106 10827 log.cc:824] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:08.298275 10828 log.cc:824] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:08.299194 10829 tablet_bootstrap.cc:654] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:08.299496 10827 tablet_bootstrap.cc:492] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62: No bootstrap required, opened a new log
I20260928 14:36:08.299706 10827 ts_tablet_manager.cc:1397] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:36:08.300299 10829 log.cc:824] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:08.300343 10828 tablet_bootstrap.cc:492] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8: No bootstrap required, opened a new log
I20260928 14:36:08.300515 10828 ts_tablet_manager.cc:1397] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:36:08.302155 10829 tablet_bootstrap.cc:492] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: No bootstrap required, opened a new log
I20260928 14:36:08.302331 10829 ts_tablet_manager.cc:1397] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260928 14:36:08.302889 10827 raft_consensus.cc:348] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.303125 10827 raft_consensus.cc:374] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:08.303176 10827 raft_consensus.cc:729] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6ec0de108f439f88f27c166aeabb62, State: Initialized, Role: FOLLOWER
I20260928 14:36:08.303617 10827 consensus_queue.cc:260] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.304255 10542 heartbeater.cc:503] Master 127.6.23.254:39827 was elected leader, sending a full tablet report...
I20260928 14:36:08.304260 10828 raft_consensus.cc:348] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.304596 10828 raft_consensus.cc:374] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:08.304661 10827 ts_tablet_manager.cc:1428] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:36:08.304983 10828 raft_consensus.cc:729] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdca84f91d1f4e9ba0cde60d4ea07dd8, State: Initialized, Role: FOLLOWER
I20260928 14:36:08.305414 10828 consensus_queue.cc:260] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.306348 10828 ts_tablet_manager.cc:1428] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 14:36:08.306344 10829 raft_consensus.cc:348] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.306609 10829 raft_consensus.cc:374] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:08.306663 10829 raft_consensus.cc:729] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff22e4422cc44b4cb6aae5ebc5ca5a3c, State: Initialized, Role: FOLLOWER
I20260928 14:36:08.306699 10676 heartbeater.cc:503] Master 127.6.23.254:39827 was elected leader, sending a full tablet report...
I20260928 14:36:08.307024 10829 consensus_queue.cc:260] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.308022 10829 ts_tablet_manager.cc:1428] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260928 14:36:08.308370 10809 heartbeater.cc:503] Master 127.6.23.254:39827 was elected leader, sending a full tablet report...
W20260928 14:36:08.484743 10810 tablet.cc:2369] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:08.495491 10834 raft_consensus.cc:482] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:08.495714 10834 raft_consensus.cc:504] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.496982 10834 leader_election.cc:288] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b6ec0de108f439f88f27c166aeabb62 (127.6.23.193:38125), ff22e4422cc44b4cb6aae5ebc5ca5a3c (127.6.23.195:42587)
I20260928 14:36:08.500818 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.497197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38125 (local address 127.6.23.194:43103)
0928 14:36:08.497339 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.497357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:08.497461 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:08.497743 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:08.497746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:08.497768 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:08.497928 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.497938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.498742 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.498745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.499689 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.499699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.500367 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.500370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.500475 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.500512 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:36:08.500587 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:36:08.500664 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:36:08.500835 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.497324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:43103 (local address 127.6.23.193:38125)
0928 14:36:08.497460 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.497464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.497483 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:36:08.497560 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.497563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.497615 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.497717 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:08.498040 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.498634 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.499807 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.500255 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.500348 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.500389 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.500610 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.500684 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:36:08.500993 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.497280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:42587 (local address 127.6.23.194:33531)
0928 14:36:08.497648 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.497664 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:08.497755 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:08.498032 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:08.498035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:08.498051 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:08.498160 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.498166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.498949 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.498952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.499866 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:08.499876 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.500594 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.500598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:08.500696 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.500718 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:36:08.500793 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:36:08.500856 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":89,"threads_started":1}
I20260928 14:36:08.501027 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.497522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:33531 (local address 127.6.23.195:42587)
0928 14:36:08.497633 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.497637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.497741 (+ 104us) server_negotiation.cc:407] Connection header received
0928 14:36:08.497851 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.497854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.497907 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.497989 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:08.498268 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.498842 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.499987 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.500460 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.500585 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.500629 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.500829 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.500905 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 14:36:08.501499 10496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e41eb9d3a742d590bdb6451bc49ee7" candidate_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6ec0de108f439f88f27c166aeabb62" is_pre_election: true
I20260928 14:36:08.501613 10763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e41eb9d3a742d590bdb6451bc49ee7" candidate_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" is_pre_election: true
I20260928 14:36:08.501822 10496 raft_consensus.cc:2454] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdca84f91d1f4e9ba0cde60d4ea07dd8 in term 0.
I20260928 14:36:08.501909 10763 raft_consensus.cc:2454] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdca84f91d1f4e9ba0cde60d4ea07dd8 in term 0.
I20260928 14:36:08.502341 10566 leader_election.cc:302] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6ec0de108f439f88f27c166aeabb62, bdca84f91d1f4e9ba0cde60d4ea07dd8; no voters:
I20260928 14:36:08.502597 10834 raft_consensus.cc:2781] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:08.502677 10834 raft_consensus.cc:482] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:08.502719 10834 raft_consensus.cc:3037] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:08.503818 10834 raft_consensus.cc:504] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.504213 10834 leader_election.cc:288] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b6ec0de108f439f88f27c166aeabb62 (127.6.23.193:38125), ff22e4422cc44b4cb6aae5ebc5ca5a3c (127.6.23.195:42587)
I20260928 14:36:08.504565 10496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e41eb9d3a742d590bdb6451bc49ee7" candidate_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6ec0de108f439f88f27c166aeabb62"
I20260928 14:36:08.504594 10763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26e41eb9d3a742d590bdb6451bc49ee7" candidate_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c"
I20260928 14:36:08.504705 10496 raft_consensus.cc:3037] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:08.504722 10763 raft_consensus.cc:3037] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:08.505837 10496 raft_consensus.cc:2454] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdca84f91d1f4e9ba0cde60d4ea07dd8 in term 1.
I20260928 14:36:08.505867 10763 raft_consensus.cc:2454] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdca84f91d1f4e9ba0cde60d4ea07dd8 in term 1.
I20260928 14:36:08.506173 10563 leader_election.cc:302] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdca84f91d1f4e9ba0cde60d4ea07dd8, ff22e4422cc44b4cb6aae5ebc5ca5a3c; no voters:
I20260928 14:36:08.506368 10834 raft_consensus.cc:2781] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:08.506593 10834 raft_consensus.cc:686] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 1 LEADER]: Becoming Leader. State: Replica: bdca84f91d1f4e9ba0cde60d4ea07dd8, State: Running, Role: LEADER
I20260928 14:36:08.506938 10834 consensus_queue.cc:237] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } }
I20260928 14:36:08.510344 10352 catalog_manager.cc:5495] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdca84f91d1f4e9ba0cde60d4ea07dd8 (127.6.23.194). New cstate: current_term: 1 leader_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdca84f91d1f4e9ba0cde60d4ea07dd8" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 42781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 } health_report { overall_health: UNKNOWN } } }
W20260928 14:36:08.518815 10677 tablet.cc:2369] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:36:08.547813 10543 tablet.cc:2369] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:08.575438 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.571778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.6.23.194:42781)
0928 14:36:08.571927 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.571931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.571944 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:36:08.572024 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.572026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.572084 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.572178 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:08.572585 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.573118 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.573977 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:08.574200 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:08.574294 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:08.574546 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:08.575002 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:08.575094 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.575149 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.575204 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:36:08.581843 10496 raft_consensus.cc:1261] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [term 1 FOLLOWER]: Refusing update from remote peer bdca84f91d1f4e9ba0cde60d4ea07dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:08.581847 10763 raft_consensus.cc:1261] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 1 FOLLOWER]: Refusing update from remote peer bdca84f91d1f4e9ba0cde60d4ea07dd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:08.582571 10834 consensus_queue.cc:1035] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff22e4422cc44b4cb6aae5ebc5ca5a3c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:08.582828 10837 consensus_queue.cc:1035] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6ec0de108f439f88f27c166aeabb62" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:08.590631 10841 mvcc.cc:205] Tried to move back new op lower bound from 7334322866503417856 to 7334322866207506432. Current Snapshot: MvccSnapshot[applied={T|T < 7334322866503417856}]
I20260928 14:36:08.592959 10840 mvcc.cc:205] Tried to move back new op lower bound from 7334322866503417856 to 7334322866207506432. Current Snapshot: MvccSnapshot[applied={T|T < 7334322866503417856}]
I20260928 14:36:08.601212 10351 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:08.601424 10351 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:08.603847 10351 catalog_manager.cc:5782] T 00000000000000000000000000000000 P faf8dee7dc724f67809afd06039d9dc4: Sending DeleteTablet for 3 replicas of tablet 26e41eb9d3a742d590bdb6451bc49ee7
I20260928 14:36:08.604679 10610 tablet_service.cc:1514] Processing DeleteTablet for tablet 26e41eb9d3a742d590bdb6451bc49ee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:08 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:35308
I20260928 14:36:08.604878 10743 tablet_service.cc:1514] Processing DeleteTablet for tablet 26e41eb9d3a742d590bdb6451bc49ee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:08 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:43324
I20260928 14:36:08.604885 10476 tablet_service.cc:1514] Processing DeleteTablet for tablet 26e41eb9d3a742d590bdb6451bc49ee7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:08 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:56992
I20260928 14:36:08.605427 10851 tablet_replica.cc:330] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62: stopping tablet replica
I20260928 14:36:08.605427 10852 tablet_replica.cc:330] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: stopping tablet replica
I20260928 14:36:08.605789 10852 raft_consensus.cc:2229] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:08.605830 10851 raft_consensus.cc:2229] T 26e41eb9d3a742d590bdb6451bc49ee7 P 0b6ec0de108f439f88f27c166aeabb62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:08.605952 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 10414
I20260928 14:36:08.607515 10852 raft_consensus.cc:2258] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:08.609153 10852 ts_tablet_manager.cc:1905] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:08.612036 10852 ts_tablet_manager.cc:1918] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:08.612195 10852 log.cc:1196] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/26e41eb9d3a742d590bdb6451bc49ee7
I20260928 14:36:08.612607 10852 ts_tablet_manager.cc:1939] T 26e41eb9d3a742d590bdb6451bc49ee7 P ff22e4422cc44b4cb6aae5ebc5ca5a3c: Deleting consensus metadata
I20260928 14:36:08.612601 10850 tablet_replica.cc:330] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8: stopping tablet replica
I20260928 14:36:08.612849 10850 raft_consensus.cc:2229] T 26e41eb9d3a742d590bdb6451bc49ee7 P bdca84f91d1f4e9ba0cde60d4ea07dd8 [term 1 LEADER]: Raft consensus shutting down.
W20260928 14:36:08.613181 10339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:38125 (error 108)
I20260928 14:36:08.613389 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 10548
I20260928 14:36:08.614831 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.614359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38125 (local address 127.0.0.1:38842)
0928 14:36:08.614627 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.614735 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:38125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260928 14:36:08.615389 10339 catalog_manager.cc:4584] TS 0b6ec0de108f439f88f27c166aeabb62 (127.6.23.193:38125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26e41eb9d3a742d590bdb6451bc49ee7: Network error: Client connection negotiation failed: client connection to 127.6.23.193:38125: connect: Connection refused (error 111)
I20260928 14:36:08.616386 10336 catalog_manager.cc:4841] TS ff22e4422cc44b4cb6aae5ebc5ca5a3c (127.6.23.195:42587): tablet 26e41eb9d3a742d590bdb6451bc49ee7 (table test-table [id=55ec4abe0d0442b4bd3a7171e3bc08d5]) successfully deleted
I20260928 14:36:08.620162 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 10681
I20260928 14:36:08.620950 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.620345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:42781 (local address 127.0.0.1:37958)
0928 14:36:08.620777 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:08.620851 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.194:42781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338}
I20260928 14:36:08.626008 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 10321
2026-09-28T14:36:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3457 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T14:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:08Z Disabled control of system clock
I20260928 14:36:08.656739 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40889
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:40889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:36:08.768901 10857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:08.769187 10857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:08.769225 10857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:08.773185 10857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:08.773240 10857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:08.773255 10857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:08.773272 10857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:08.773286 10857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:08.777979 10857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:40889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:08.779024 10857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:08.780385 10857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:08.789134 10862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:08.789148 10863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:08.789175 10865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:08.789597 10857 server_base.cc:1034] running on GCE node
I20260928 14:36:08.790191 10857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:08.791339 10857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:08.792522 10857 hybrid_clock.cc:648] HybridClock initialized: now 1790606168792501 us; error 38 us; skew 500 ppm
I20260928 14:36:08.794349 10857 webserver.cc:460] Webserver started at http://127.6.23.254:46765/ using document root <none> and password file <none>
I20260928 14:36:08.794873 10857 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:08.794931 10857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:08.795094 10857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:08.797111 10857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "f233309eb86a46708fdb96d12c1ad2ca"
format_stamp: "Formatted at 2026-09-28 14:36:08 on dist-test-slave-06lh"
server_key: "de95f2ecb29937b2e1adeda5e03b11bb"
server_key_iv: "52ee930c5a28145ffcfbcab34da5da98"
server_key_version: "encryptionkey@0"
I20260928 14:36:08.797616 10857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "f233309eb86a46708fdb96d12c1ad2ca"
format_stamp: "Formatted at 2026-09-28 14:36:08 on dist-test-slave-06lh"
server_key: "de95f2ecb29937b2e1adeda5e03b11bb"
server_key_iv: "52ee930c5a28145ffcfbcab34da5da98"
server_key_version: "encryptionkey@0"
I20260928 14:36:08.801221 10857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 14:36:08.803635 10871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:08.804888 10857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:08.805028 10857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "f233309eb86a46708fdb96d12c1ad2ca"
format_stamp: "Formatted at 2026-09-28 14:36:08 on dist-test-slave-06lh"
server_key: "de95f2ecb29937b2e1adeda5e03b11bb"
server_key_iv: "52ee930c5a28145ffcfbcab34da5da98"
server_key_version: "encryptionkey@0"
I20260928 14:36:08.805142 10857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:08.809685 10857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:08.810283 10857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:08.810453 10857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:08.817341 10857 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:40889
I20260928 14:36:08.817348 10923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:40889 every 8 connection(s)
I20260928 14:36:08.818248 10857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:08.821467 10924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:08.821962 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 10857
I20260928 14:36:08.822152 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:08.822471 6239 external_mini_cluster.cc:1428] Setting key f4bfd8c698b31d98cb87c78fca113b91
I20260928 14:36:08.827857 10924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca: Bootstrap starting.
I20260928 14:36:08.828035 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:08.823868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50418 (local address 127.6.23.254:40889)
0928 14:36:08.824544 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:08.824557 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:08.824590 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:36:08.825238 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:08.825255 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:08.825600 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:08.825925 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:08.826707 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:08.826712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:08.826723 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:08.826733 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:08.826786 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:08.826801 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:08.826947 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:08.827018 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 14:36:08.827335 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":125,"threads_started":1}
I20260928 14:36:08.830338 10924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:08.831061 10924 log.cc:824] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:08.832939 10924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca: No bootstrap required, opened a new log
I20260928 14:36:08.836583 10924 raft_consensus.cc:348] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f233309eb86a46708fdb96d12c1ad2ca" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40889 } }
I20260928 14:36:08.836827 10924 raft_consensus.cc:374] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:08.836912 10924 raft_consensus.cc:729] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f233309eb86a46708fdb96d12c1ad2ca, State: Initialized, Role: FOLLOWER
I20260928 14:36:08.837345 10924 consensus_queue.cc:260] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f233309eb86a46708fdb96d12c1ad2ca" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40889 } }
I20260928 14:36:08.837510 10924 raft_consensus.cc:388] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:08.837580 10924 raft_consensus.cc:482] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:08.837714 10924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:08.838852 10924 raft_consensus.cc:504] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f233309eb86a46708fdb96d12c1ad2ca" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40889 } }
I20260928 14:36:08.839151 10924 leader_election.cc:302] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f233309eb86a46708fdb96d12c1ad2ca; no voters:
I20260928 14:36:08.839502 10924 leader_election.cc:288] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:08.839630 10929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:08.839875 10929 raft_consensus.cc:686] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [term 1 LEADER]: Becoming Leader. State: Replica: f233309eb86a46708fdb96d12c1ad2ca, State: Running, Role: LEADER
I20260928 14:36:08.840215 10929 consensus_queue.cc:237] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f233309eb86a46708fdb96d12c1ad2ca" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40889 } }
I20260928 14:36:08.840776 10924 sys_catalog.cc:564] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:08.841933 10930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f233309eb86a46708fdb96d12c1ad2ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f233309eb86a46708fdb96d12c1ad2ca" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40889 } } }
I20260928 14:36:08.842054 10930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:08.842408 10938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:08.842437 10931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f233309eb86a46708fdb96d12c1ad2ca. Latest consensus state: current_term: 1 leader_uuid: "f233309eb86a46708fdb96d12c1ad2ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f233309eb86a46708fdb96d12c1ad2ca" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40889 } } }
I20260928 14:36:08.843106 10931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:08.845618 10938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:08.851293 10938 catalog_manager.cc:1347] Generated new cluster ID: 396fb3841d8342d3aef08f9cc5c8a4c6
I20260928 14:36:08.851357 10938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:08.873683 10938 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:08.874050 10938 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:08.880537 10938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca: Generated new TSK 0
I20260928 14:36:08.881362 10938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:36:08.893147 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40889
--builtin_ntp_servers=127.6.23.212:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:36:09.005113 10948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:09.005350 10948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:09.005388 10948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:09.009456 10948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:09.009502 10948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:09.009560 10948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:09.014070 10948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:09.015026 10948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:09.016316 10948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:09.025702 10953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:09.025702 10954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:09.025708 10956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:09.026108 10948 server_base.cc:1034] running on GCE node
I20260928 14:36:09.026512 10948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:09.027143 10948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:09.028297 10948 hybrid_clock.cc:648] HybridClock initialized: now 1790606169028279 us; error 28 us; skew 500 ppm
I20260928 14:36:09.030150 10948 webserver.cc:460] Webserver started at http://127.6.23.193:40065/ using document root <none> and password file <none>
I20260928 14:36:09.030735 10948 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:09.030795 10948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:09.031056 10948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:09.033159 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "aed6e44c2c5944ec9d34a58a65ca70a2"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "d05c18a32b7b036ab1124d3deae358df"
server_key_iv: "ffb0aeb9305797f7dc4ca65ad031a599"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.033730 10948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "aed6e44c2c5944ec9d34a58a65ca70a2"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "d05c18a32b7b036ab1124d3deae358df"
server_key_iv: "ffb0aeb9305797f7dc4ca65ad031a599"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.037465 10948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 14:36:09.039949 10962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.041271 10948 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:36:09.041422 10948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "aed6e44c2c5944ec9d34a58a65ca70a2"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "d05c18a32b7b036ab1124d3deae358df"
server_key_iv: "ffb0aeb9305797f7dc4ca65ad031a599"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.041565 10948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:09.046347 10948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:09.046949 10948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:09.047142 10948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:09.047777 10948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:09.048709 10948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:09.048782 10948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.048867 10948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:09.048921 10948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.058734 10948 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:38497
I20260928 14:36:09.058756 11075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:38497 every 8 connection(s)
I20260928 14:36:09.059806 10948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:09.066076 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.061734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:60863 (local address 127.6.23.254:40889)
0928 14:36:09.061902 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.061907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.063045 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 14:36:09.064418 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.064421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.064459 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.064545 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.065644 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.065647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.065649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.065661 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.065681 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.065689 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.065768 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.065928 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.065969 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:36:09.066948 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.062130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40889 (local address 127.6.23.193:60863)
0928 14:36:09.062874 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.062929 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:09.064197 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:09.064712 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:09.064719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:09.064809 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:09.065358 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:09.065359 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:09.065421 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:09.065491 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:09.065768 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:09.065775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:09.065779 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:36:09.065910 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 14:36:09.066104 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":137,"threads_started":1}
I20260928 14:36:09.068219 11076 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40889
I20260928 14:36:09.068449 11076 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:09.068864 11076 heartbeater.cc:511] Master 127.6.23.254:40889 requested a full tablet report, sending...
I20260928 14:36:09.069649 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 10948
I20260928 14:36:09.069782 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:09.070055 6239 external_mini_cluster.cc:1428] Setting key fa763289015129409b386717c0c972f5
I20260928 14:36:09.070480 10888 ts_manager.cc:194] Registered new tserver with Master: aed6e44c2c5944ec9d34a58a65ca70a2 (127.6.23.193:38497)
I20260928 14:36:09.072146 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40889
--builtin_ntp_servers=127.6.23.212:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:36:09.184731 11079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:09.185006 11079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:09.185046 11079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:09.189253 11079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:09.189323 11079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:09.189389 11079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:09.194300 11079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:09.195562 11079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:09.196971 11079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:09.206781 11085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:09.207069 11079 server_base.cc:1034] running on GCE node
W20260928 14:36:09.207091 11087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:09.207329 11084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:09.207770 11079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:09.208456 11079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:09.209636 11079 hybrid_clock.cc:648] HybridClock initialized: now 1790606169209616 us; error 50 us; skew 500 ppm
I20260928 14:36:09.211519 11079 webserver.cc:460] Webserver started at http://127.6.23.194:45317/ using document root <none> and password file <none>
I20260928 14:36:09.212107 11079 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:09.212200 11079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:09.212443 11079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:09.214507 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "23537c94e7d942f3a9a97375d41d3d99"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "9da45442fe342d919021df760569db30"
server_key_iv: "607ac1301133804e6424a3fad4cae10d"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.215098 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "23537c94e7d942f3a9a97375d41d3d99"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "9da45442fe342d919021df760569db30"
server_key_iv: "607ac1301133804e6424a3fad4cae10d"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.218915 11079 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:36:09.221530 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.222946 11079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:36:09.223094 11079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "23537c94e7d942f3a9a97375d41d3d99"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "9da45442fe342d919021df760569db30"
server_key_iv: "607ac1301133804e6424a3fad4cae10d"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.223218 11079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:09.228020 11079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:09.228652 11079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:09.228839 11079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:09.229399 11079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:09.230671 11079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:09.230726 11079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.230787 11079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:09.230816 11079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.240048 11079 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:44321
I20260928 14:36:09.240072 11206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:44321 every 8 connection(s)
I20260928 14:36:09.241068 11079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:09.247478 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.243018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:40437 (local address 127.6.23.254:40889)
0928 14:36:09.243189 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.243193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.244356 (+ 1163us) server_negotiation.cc:407] Connection header received
0928 14:36:09.245785 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.245788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.245826 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.245903 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.246999 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.247003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.247005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.247017 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.247036 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.247044 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.247125 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.247319 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.247363 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:36:09.248440 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.243516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40889 (local address 127.6.23.194:40437)
0928 14:36:09.244206 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.244246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:09.245555 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:09.246067 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:09.246073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:09.246165 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:09.246748 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:09.246750 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:09.246790 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:09.246858 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:09.247121 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:09.247127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:09.247131 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:36:09.247326 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 14:36:09.247532 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
I20260928 14:36:09.248467 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 11079
I20260928 14:36:09.248584 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:09.248921 6239 external_mini_cluster.cc:1428] Setting key b78e7e68d41e07bbba0bf55c2f43f11a
I20260928 14:36:09.249814 11207 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40889
I20260928 14:36:09.250049 11207 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:09.250474 11207 heartbeater.cc:511] Master 127.6.23.254:40889 requested a full tablet report, sending...
I20260928 14:36:09.251070 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40889
--builtin_ntp_servers=127.6.23.212:38993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:36:09.251543 10888 ts_manager.cc:194] Registered new tserver with Master: 23537c94e7d942f3a9a97375d41d3d99 (127.6.23.194:44321)
W20260928 14:36:09.363022 11210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:09.363368 11210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:09.363415 11210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:09.367522 11210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:09.367579 11210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:09.367637 11210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:09.372434 11210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:09.373483 11210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:09.374828 11210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:36:09.384971 11210 server_base.cc:1034] running on GCE node
W20260928 14:36:09.384933 11216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:09.384934 11215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:09.384941 11218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:09.385553 11210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:09.386245 11210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:09.387426 11210 hybrid_clock.cc:648] HybridClock initialized: now 1790606169387391 us; error 52 us; skew 500 ppm
I20260928 14:36:09.389215 11210 webserver.cc:460] Webserver started at http://127.6.23.195:33909/ using document root <none> and password file <none>
I20260928 14:36:09.389809 11210 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:09.389900 11210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:09.390130 11210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:09.392205 11210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "5bb48de7b08cb94cdf22e6457461529a"
server_key_iv: "e243f76dfefa21ad715a2c2e265487fa"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.392764 11210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "5bb48de7b08cb94cdf22e6457461529a"
server_key_iv: "e243f76dfefa21ad715a2c2e265487fa"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.396351 11210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 14:36:09.398693 11224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.399923 11210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:09.400055 11210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d"
format_stamp: "Formatted at 2026-09-28 14:36:09 on dist-test-slave-06lh"
server_key: "5bb48de7b08cb94cdf22e6457461529a"
server_key_iv: "e243f76dfefa21ad715a2c2e265487fa"
server_key_version: "encryptionkey@0"
I20260928 14:36:09.400166 11210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:09.404611 11210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:09.405189 11210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:09.405367 11210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:09.405889 11210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:09.406754 11210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:09.406821 11210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.406888 11210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:09.406939 11210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:09.415767 11210 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:38207
I20260928 14:36:09.415843 11337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:38207 every 8 connection(s)
I20260928 14:36:09.416705 11210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:09.417403 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 11210
I20260928 14:36:09.417554 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:09.417908 6239 external_mini_cluster.cc:1428] Setting key 719ea7cd9aa69366f508cc6f5e4b78b0
I20260928 14:36:09.422979 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.418649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:36819 (local address 127.6.23.254:40889)
0928 14:36:09.418800 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.418804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.420214 (+ 1410us) server_negotiation.cc:407] Connection header received
0928 14:36:09.421480 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.421482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.421518 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.421592 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.422597 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.422601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.422603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.422615 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.422641 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.422651 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.422745 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.422843 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.422880 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:36:09.423873 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.419167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40889 (local address 127.6.23.195:36819)
0928 14:36:09.420047 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.420103 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:09.421287 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:09.421722 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:09.421728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:09.421813 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:09.422325 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:09.422326 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:09.422371 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:09.422457 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:09.422732 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:09.422736 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:09.422739 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:36:09.422856 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 14:36:09.423082 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":131,"threads_started":1}
I20260928 14:36:09.425050 11338 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40889
I20260928 14:36:09.425283 11338 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:09.425690 11338 heartbeater.cc:511] Master 127.6.23.254:40889 requested a full tablet report, sending...
I20260928 14:36:09.426786 10888 ts_manager.cc:194] Registered new tserver with Master: 5ddc4a3a1d8445c2af6469e5ac4ee62d (127.6.23.195:38207)
I20260928 14:36:09.431877 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:36:09.435595 10927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.433774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.6.23.254:40889)
0928 14:36:09.433924 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.433932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.434100 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:36:09.434263 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.434266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.434307 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.434428 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.435145 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.435149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.435151 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.435162 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.435182 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.435191 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.435327 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.435413 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.435455 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:36:09.438148 10888 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:36:09.441273 10888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:09.453326 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.449735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38497 (local address 127.0.0.1:43556)
0928 14:36:09.450588 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.450605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:09.450678 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:09.451796 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:09.451799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:09.451809 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:09.452538 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:09.452540 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:09.452560 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:09.452630 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:09.452965 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:09.452976 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:09.452982 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:36:09.453095 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:36:09.453169 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":72,"threads_started":1}
I20260928 14:36:09.453421 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.450855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.6.23.193:38497)
0928 14:36:09.451436 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.451447 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.451467 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:36:09.451517 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.451522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.451665 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.452060 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.452747 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.452755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.452771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.452784 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.452829 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.452873 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.452979 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.453152 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.453230 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":136,"threads_started":1}
I20260928 14:36:09.453611 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.449728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:38207 (local address 127.0.0.1:39186)
0928 14:36:09.450496 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.450549 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:09.450655 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:09.451692 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:09.451698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:09.451767 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:09.452689 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:09.452691 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:09.452702 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:09.452730 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:09.453224 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:09.453227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:09.453230 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:36:09.453293 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:36:09.453321 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":104,"threads_started":1}
I20260928 14:36:09.453946 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.449983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:44321 (local address 127.0.0.1:38648)
0928 14:36:09.450835 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.450844 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:09.450924 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:09.451681 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:09.451687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:09.451765 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:09.452745 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:09.452747 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:09.452757 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:09.452776 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:09.453055 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:09.453058 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:09.453061 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:36:09.453292 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 14:36:09.453330 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":93,"threads_started":1}
I20260928 14:36:09.454110 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.449845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.6.23.195:38207)
0928 14:36:09.451169 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.451186 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.451207 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:36:09.451314 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.451319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.451536 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.452401 (+ 865us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.452993 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.452999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.453009 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.453027 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.453068 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.453094 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.453186 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.453381 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.453926 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1215,"thread_start_us":144,"threads_started":1}
I20260928 14:36:09.454433 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.450496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.6.23.194:44321)
0928 14:36:09.451329 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.451338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.451358 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:36:09.451402 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.451406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.451537 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.451685 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.452874 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.452880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.452891 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.452906 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.452949 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.452978 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.453076 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.454195 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.454262 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":116,"threads_started":1}
I20260928 14:36:09.456481 11010 tablet_service.cc:1467] Processing CreateTablet for tablet 991fe72ad0db40bfbe73844f2c9b518b (DEFAULT_TABLE table=test-table [id=a68c5059a34942febda42552abc045a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:09.457072 11272 tablet_service.cc:1467] Processing CreateTablet for tablet 991fe72ad0db40bfbe73844f2c9b518b (DEFAULT_TABLE table=test-table [id=a68c5059a34942febda42552abc045a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:09.457056 11141 tablet_service.cc:1467] Processing CreateTablet for tablet 991fe72ad0db40bfbe73844f2c9b518b (DEFAULT_TABLE table=test-table [id=a68c5059a34942febda42552abc045a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:09.457623 11010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991fe72ad0db40bfbe73844f2c9b518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:09.458053 11141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991fe72ad0db40bfbe73844f2c9b518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:09.458053 11272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 991fe72ad0db40bfbe73844f2c9b518b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:09.463817 11355 tablet_bootstrap.cc:492] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2: Bootstrap starting.
I20260928 14:36:09.465569 11357 tablet_bootstrap.cc:492] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Bootstrap starting.
I20260928 14:36:09.466141 11355 tablet_bootstrap.cc:654] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:09.466943 11355 log.cc:824] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:09.467574 11356 tablet_bootstrap.cc:492] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Bootstrap starting.
I20260928 14:36:09.467751 11357 tablet_bootstrap.cc:654] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:09.468559 11357 log.cc:824] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:09.468916 11355 tablet_bootstrap.cc:492] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2: No bootstrap required, opened a new log
I20260928 14:36:09.469128 11355 ts_tablet_manager.cc:1397] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 14:36:09.469702 11356 tablet_bootstrap.cc:654] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:09.470520 11357 tablet_bootstrap.cc:492] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: No bootstrap required, opened a new log
I20260928 14:36:09.470480 11356 log.cc:824] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:09.470700 11357 ts_tablet_manager.cc:1397] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:36:09.472419 11356 tablet_bootstrap.cc:492] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: No bootstrap required, opened a new log
I20260928 14:36:09.472604 11356 ts_tablet_manager.cc:1397] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:36:09.473340 11355 raft_consensus.cc:348] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.473589 11355 raft_consensus.cc:374] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:09.473649 11355 raft_consensus.cc:729] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed6e44c2c5944ec9d34a58a65ca70a2, State: Initialized, Role: FOLLOWER
I20260928 14:36:09.474107 11355 consensus_queue.cc:260] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.474781 11076 heartbeater.cc:503] Master 127.6.23.254:40889 was elected leader, sending a full tablet report...
I20260928 14:36:09.475096 11355 ts_tablet_manager.cc:1428] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 14:36:09.475889 11357 raft_consensus.cc:348] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.476138 11357 raft_consensus.cc:374] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:09.476198 11357 raft_consensus.cc:729] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23537c94e7d942f3a9a97375d41d3d99, State: Initialized, Role: FOLLOWER
I20260928 14:36:09.476639 11357 consensus_queue.cc:260] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.477078 11356 raft_consensus.cc:348] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.477316 11356 raft_consensus.cc:374] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:09.477368 11356 raft_consensus.cc:729] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddc4a3a1d8445c2af6469e5ac4ee62d, State: Initialized, Role: FOLLOWER
I20260928 14:36:09.477615 11357 ts_tablet_manager.cc:1428] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 14:36:09.477761 11356 consensus_queue.cc:260] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.477985 11207 heartbeater.cc:503] Master 127.6.23.254:40889 was elected leader, sending a full tablet report...
I20260928 14:36:09.478663 11356 ts_tablet_manager.cc:1428] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:36:09.478971 11338 heartbeater.cc:503] Master 127.6.23.254:40889 was elected leader, sending a full tablet report...
W20260928 14:36:09.492026 11208 tablet.cc:2369] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:36:09.560993 11077 tablet.cc:2369] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:36:09.667757 11339 tablet.cc:2369] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:09.870263 11361 raft_consensus.cc:482] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:09.870558 11361 raft_consensus.cc:504] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.871834 11361 leader_election.cc:288] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ddc4a3a1d8445c2af6469e5ac4ee62d (127.6.23.195:38207), 23537c94e7d942f3a9a97375d41d3d99 (127.6.23.194:44321)
I20260928 14:36:09.874284 11351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.872370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:54043 (local address 127.6.23.195:38207)
0928 14:36:09.872524 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.872529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.872798 (+ 269us) server_negotiation.cc:407] Connection header received
0928 14:36:09.872952 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.872955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.872998 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.873099 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.873855 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.873859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.873863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.873877 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.873901 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.873913 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.874008 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.874076 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.874135 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:36:09.874315 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.872154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:38207 (local address 127.6.23.193:54043)
0928 14:36:09.872658 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.872669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:09.872821 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:09.873143 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:09.873146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:09.873160 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:09.873706 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:09.873708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:09.873717 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:09.873739 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:09.873997 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:09.874000 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:09.874003 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:36:09.874090 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:36:09.874129 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":50,"spinlock_wait_cycles":11904,"thread_start_us":83,"threads_started":1}
I20260928 14:36:09.874394 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.872258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:41435 (local address 127.6.23.194:44321)
0928 14:36:09.872408 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.872413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.872752 (+ 339us) server_negotiation.cc:407] Connection header received
0928 14:36:09.872928 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.872932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.872971 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.873049 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.873985 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.873989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.873991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.874003 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.874034 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.874048 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.874134 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.874225 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.874261 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:36:09.874454 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.872154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:44321 (local address 127.6.23.193:41435)
0928 14:36:09.872636 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.872653 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:09.872813 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:09.873098 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:09.873101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:09.873117 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:09.873796 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:09.873798 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:09.873809 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:09.873844 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:09.874137 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:09.874140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:09.874143 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:36:09.874224 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:36:09.874255 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"spinlock_wait_cycles":22016,"thread_start_us":101,"threads_started":1}
I20260928 14:36:09.875105 11292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991fe72ad0db40bfbe73844f2c9b518b" candidate_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" is_pre_election: true
I20260928 14:36:09.875121 11161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991fe72ad0db40bfbe73844f2c9b518b" candidate_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23537c94e7d942f3a9a97375d41d3d99" is_pre_election: true
I20260928 14:36:09.875437 11161 raft_consensus.cc:2454] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aed6e44c2c5944ec9d34a58a65ca70a2 in term 0.
I20260928 14:36:09.875445 11292 raft_consensus.cc:2454] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aed6e44c2c5944ec9d34a58a65ca70a2 in term 0.
I20260928 14:36:09.875962 10964 leader_election.cc:302] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23537c94e7d942f3a9a97375d41d3d99, 5ddc4a3a1d8445c2af6469e5ac4ee62d, aed6e44c2c5944ec9d34a58a65ca70a2; no voters:
I20260928 14:36:09.876240 11361 raft_consensus.cc:2781] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:09.876312 11361 raft_consensus.cc:482] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:09.876343 11361 raft_consensus.cc:3037] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:09.877388 11361 raft_consensus.cc:504] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.877768 11361 leader_election.cc:288] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [CANDIDATE]: Term 1 election: Requested vote from peers 5ddc4a3a1d8445c2af6469e5ac4ee62d (127.6.23.195:38207), 23537c94e7d942f3a9a97375d41d3d99 (127.6.23.194:44321)
I20260928 14:36:09.878114 11292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991fe72ad0db40bfbe73844f2c9b518b" candidate_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d"
I20260928 14:36:09.878149 11161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "991fe72ad0db40bfbe73844f2c9b518b" candidate_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23537c94e7d942f3a9a97375d41d3d99"
I20260928 14:36:09.878237 11292 raft_consensus.cc:3037] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:09.878284 11161 raft_consensus.cc:3037] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:09.879381 11292 raft_consensus.cc:2454] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aed6e44c2c5944ec9d34a58a65ca70a2 in term 1.
I20260928 14:36:09.879446 11161 raft_consensus.cc:2454] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aed6e44c2c5944ec9d34a58a65ca70a2 in term 1.
I20260928 14:36:09.879709 10965 leader_election.cc:302] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ddc4a3a1d8445c2af6469e5ac4ee62d, aed6e44c2c5944ec9d34a58a65ca70a2; no voters:
I20260928 14:36:09.879863 11361 raft_consensus.cc:2781] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:09.880045 11361 raft_consensus.cc:686] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [term 1 LEADER]: Becoming Leader. State: Replica: aed6e44c2c5944ec9d34a58a65ca70a2, State: Running, Role: LEADER
I20260928 14:36:09.880363 11361 consensus_queue.cc:237] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } }
I20260928 14:36:09.883786 10886 catalog_manager.cc:5495] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to aed6e44c2c5944ec9d34a58a65ca70a2 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed6e44c2c5944ec9d34a58a65ca70a2" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 } health_report { overall_health: UNKNOWN } } }
I20260928 14:36:09.904671 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.903124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43572 (local address 127.6.23.193:38497)
0928 14:36:09.903313 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:09.903318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:09.903338 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:36:09.903411 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:09.903413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:09.903470 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:09.903570 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:09.904265 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:09.904268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:09.904271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:09.904289 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:09.904307 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:09.904315 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:09.904407 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:09.904478 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 14:36:09.904524 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:36:09.910590 11161 raft_consensus.cc:1261] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 1 FOLLOWER]: Refusing update from remote peer aed6e44c2c5944ec9d34a58a65ca70a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:09.910591 11292 raft_consensus.cc:1261] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 1 FOLLOWER]: Refusing update from remote peer aed6e44c2c5944ec9d34a58a65ca70a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:09.911340 11361 consensus_queue.cc:1035] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23537c94e7d942f3a9a97375d41d3d99" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:09.911586 11366 consensus_queue.cc:1035] T 991fe72ad0db40bfbe73844f2c9b518b P aed6e44c2c5944ec9d34a58a65ca70a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ddc4a3a1d8445c2af6469e5ac4ee62d" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:09.919185 11369 mvcc.cc:205] Tried to move back new op lower bound from 7334322871946641408 to 7334322871832518656. Current Snapshot: MvccSnapshot[applied={T|T < 7334322871946641408}]
I20260928 14:36:09.919478 11370 mvcc.cc:205] Tried to move back new op lower bound from 7334322871946641408 to 7334322871832518656. Current Snapshot: MvccSnapshot[applied={T|T < 7334322871946641408}]
I20260928 14:36:09.920935 11371 mvcc.cc:205] Tried to move back new op lower bound from 7334322871946641408 to 7334322871832518656. Current Snapshot: MvccSnapshot[applied={T|T < 7334322871946641408}]
I20260928 14:36:09.928484 10888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:09.928645 10888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:09.931015 10888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f233309eb86a46708fdb96d12c1ad2ca: Sending DeleteTablet for 3 replicas of tablet 991fe72ad0db40bfbe73844f2c9b518b
I20260928 14:36:09.931851 11010 tablet_service.cc:1514] Processing DeleteTablet for tablet 991fe72ad0db40bfbe73844f2c9b518b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:09 UTC) from {username='slave'} at 127.0.0.1:43556
I20260928 14:36:09.931854 11141 tablet_service.cc:1514] Processing DeleteTablet for tablet 991fe72ad0db40bfbe73844f2c9b518b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:09 UTC) from {username='slave'} at 127.0.0.1:38648
I20260928 14:36:09.931860 11272 tablet_service.cc:1514] Processing DeleteTablet for tablet 991fe72ad0db40bfbe73844f2c9b518b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:09 UTC) from {username='slave'} at 127.0.0.1:39186
I20260928 14:36:09.932482 11378 tablet_replica.cc:330] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: stopping tablet replica
I20260928 14:36:09.932790 11378 raft_consensus.cc:2229] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:09.933171 11378 raft_consensus.cc:2258] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:09.933284 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 10948
I20260928 14:36:09.935155 11379 tablet_replica.cc:330] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: stopping tablet replica
I20260928 14:36:09.935478 11379 raft_consensus.cc:2229] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:09.935719 11379 raft_consensus.cc:2258] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:09.936777 11378 ts_tablet_manager.cc:1905] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:09.937247 11379 ts_tablet_manager.cc:1905] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:09.939040 11378 ts_tablet_manager.cc:1918] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:09.939155 11378 log.cc:1196] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/991fe72ad0db40bfbe73844f2c9b518b
I20260928 14:36:09.939494 11378 ts_tablet_manager.cc:1939] T 991fe72ad0db40bfbe73844f2c9b518b P 5ddc4a3a1d8445c2af6469e5ac4ee62d: Deleting consensus metadata
I20260928 14:36:09.940706 11379 ts_tablet_manager.cc:1918] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:09.940690 10874 catalog_manager.cc:4841] TS 5ddc4a3a1d8445c2af6469e5ac4ee62d (127.6.23.195:38207): tablet 991fe72ad0db40bfbe73844f2c9b518b (table test-table [id=a68c5059a34942febda42552abc045a9]) successfully deleted
I20260928 14:36:09.940804 11379 log.cc:1196] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/991fe72ad0db40bfbe73844f2c9b518b
I20260928 14:36:09.941131 11379 ts_tablet_manager.cc:1939] T 991fe72ad0db40bfbe73844f2c9b518b P 23537c94e7d942f3a9a97375d41d3d99: Deleting consensus metadata
W20260928 14:36:09.941179 10872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:38497 (error 108)
I20260928 14:36:09.942171 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:09.941874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38497 (local address 127.0.0.1:43584)
0928 14:36:09.942026 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:09.942099 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:38497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 14:36:09.942318 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 11079
I20260928 14:36:09.942456 10873 catalog_manager.cc:4841] TS 23537c94e7d942f3a9a97375d41d3d99 (127.6.23.194:44321): tablet 991fe72ad0db40bfbe73844f2c9b518b (table test-table [id=a68c5059a34942febda42552abc045a9]) successfully deleted
W20260928 14:36:09.943022 10872 catalog_manager.cc:4584] TS aed6e44c2c5944ec9d34a58a65ca70a2 (127.6.23.193:38497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 991fe72ad0db40bfbe73844f2c9b518b: Network error: Client connection negotiation failed: client connection to 127.6.23.193:38497: connect: Connection refused (error 111)
I20260928 14:36:09.948302 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 11210
I20260928 14:36:09.953794 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 10857
2026-09-28T14:36:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1326 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 14:36:09.975896 6239 mini_oidc.cc:152] Starting JWKS server
I20260928 14:36:09.977447 6239 webserver.cc:460] Webserver started at https://127.0.0.1:40715/ using document root <none> and password file <none>
I20260928 14:36:09.977578 6239 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:36:09.977789 6239 webserver.cc:460] Webserver started at http://127.0.0.1:43763/ using document root <none> and password file <none>
2026-09-28T14:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:09Z Disabled control of system clock
I20260928 14:36:09.986989 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45677
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45677
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40715/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/testchainca.cert with env {}
W20260928 14:36:10.097635 11387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:10.097892 11387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:10.097965 11387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:10.101997 11387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:10.102054 11387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:10.102070 11387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:10.102087 11387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:10.102154 11387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:36:10.102176 11387 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:36:10.102205 11387 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40715/jwks.json
I20260928 14:36:10.108160 11387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40715/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:10.109231 11387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:10.110245 11387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:10.118820 11392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:10.118822 11393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:10.118891 11395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:10.119140 11387 server_base.cc:1034] running on GCE node
I20260928 14:36:10.119831 11387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:10.120970 11387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:10.122165 11387 hybrid_clock.cc:648] HybridClock initialized: now 1790606170122152 us; error 27 us; skew 500 ppm
I20260928 14:36:10.124162 11387 webserver.cc:460] Webserver started at http://127.6.23.254:44887/ using document root <none> and password file <none>
I20260928 14:36:10.124778 11387 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:10.124874 11387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:10.125092 11387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:10.127179 11387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "bb23e8a0ee2b4c318eef41e73004c5d7"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "c2b446afa7f8764759d302f5620f3d6e"
server_key_iv: "cda597dab3a72d48d67fe60d181ba0a9"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.127839 11387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "bb23e8a0ee2b4c318eef41e73004c5d7"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "c2b446afa7f8764759d302f5620f3d6e"
server_key_iv: "cda597dab3a72d48d67fe60d181ba0a9"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.131695 11387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:36:10.134197 11401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:10.135699 11387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:10.135856 11387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "bb23e8a0ee2b4c318eef41e73004c5d7"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "c2b446afa7f8764759d302f5620f3d6e"
server_key_iv: "cda597dab3a72d48d67fe60d181ba0a9"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.136003 11387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:10.150990 11387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:10.151840 11387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:10.152063 11387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:10.159605 11387 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:45677
I20260928 14:36:10.159615 11453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:45677 every 8 connection(s)
I20260928 14:36:10.160609 11387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:10.162397 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 11387
I20260928 14:36:10.162532 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:10.162861 6239 external_mini_cluster.cc:1428] Setting key e89e6c858dd25c6d73f928df48251744
I20260928 14:36:10.164776 11454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:10.171543 11454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7: Bootstrap starting.
I20260928 14:36:10.172173 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:10.164210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.6.23.254:45677)
0928 14:36:10.164847 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:10.164871 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:10.164921 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:36:10.165596 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:10.165613 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:10.166007 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:10.166287 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:10.166626 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.167678 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.168860 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.169343 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.169458 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.170345 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:10.170350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:10.170362 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:10.170765 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:10.170823 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:10.170844 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:10.170918 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:10.171087 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:36:10.171413 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":134,"threads_started":1}
I20260928 14:36:10.174014 11454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:10.174925 11454 log.cc:824] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:10.175107 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:36:10.177333 11454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7: No bootstrap required, opened a new log
I20260928 14:36:10.181232 11454 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb23e8a0ee2b4c318eef41e73004c5d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45677 } }
I20260928 14:36:10.181463 11454 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:10.181528 11454 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb23e8a0ee2b4c318eef41e73004c5d7, State: Initialized, Role: FOLLOWER
I20260928 14:36:10.181980 11454 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb23e8a0ee2b4c318eef41e73004c5d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45677 } }
I20260928 14:36:10.182121 11454 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:10.182186 11454 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:10.182303 11454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:10.183378 11454 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb23e8a0ee2b4c318eef41e73004c5d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45677 } }
I20260928 14:36:10.183749 11454 leader_election.cc:302] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb23e8a0ee2b4c318eef41e73004c5d7; no voters:
I20260928 14:36:10.184098 11454 leader_election.cc:288] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:10.184324 11459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:10.184582 11459 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [term 1 LEADER]: Becoming Leader. State: Replica: bb23e8a0ee2b4c318eef41e73004c5d7, State: Running, Role: LEADER
I20260928 14:36:10.184878 11459 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb23e8a0ee2b4c318eef41e73004c5d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45677 } }
I20260928 14:36:10.185390 11454 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:10.186612 11461 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb23e8a0ee2b4c318eef41e73004c5d7. Latest consensus state: current_term: 1 leader_uuid: "bb23e8a0ee2b4c318eef41e73004c5d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb23e8a0ee2b4c318eef41e73004c5d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45677 } } }
I20260928 14:36:10.186785 11461 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:10.186615 11460 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb23e8a0ee2b4c318eef41e73004c5d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb23e8a0ee2b4c318eef41e73004c5d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45677 } } }
I20260928 14:36:10.187019 11460 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:10.187503 11468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:10.190121 11468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:10.195083 11468 catalog_manager.cc:1347] Generated new cluster ID: ab07f708494d4befa7a7b7a6ca59e0cc
I20260928 14:36:10.195152 11468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:10.200337 11468 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:10.201545 11468 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:10.206348 11468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb23e8a0ee2b4c318eef41e73004c5d7: Generated new TSK 0
I20260928 14:36:10.207131 11468 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:36:10.242667 11484 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 14:36:10.242802 11484 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:36:10.243430 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:10.225629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43510 (local address 127.6.23.254:45677)
0928 14:36:10.225753 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:10.225757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:10.226011 (+ 254us) server_negotiation.cc:407] Connection header received
0928 14:36:10.226255 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:10.226257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:10.226308 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:10.226394 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:36:10.226693 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.227284 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.228557 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.228768 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.228840 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.229221 (+ 381us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:36:10.242992 (+ 13771us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:36:10.243308 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43510: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 14:36:10.243515 11457 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43510: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 14:36:10.244935 11483 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:10.225455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45677 (local address 127.0.0.1:43510)
0928 14:36:10.225866 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:10.225888 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:10.226053 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:10.226440 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:10.226444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:10.226452 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:36:10.226586 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:10.226593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.227413 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.227417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:10.228443 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:10.228454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.228880 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.228884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:10.228979 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.229036 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:36:10.243272 (+ 14236us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:36:10.243573 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.23.254:45677: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.23.254:45677: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40715/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 14:36:10.245769 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 11387
2026-09-28T14:36:10Z chronyd exiting
I20260928 14:36:10.282985 6239 test_util.cc:175] -----------------------------------------------
I20260928 14:36:10.283077 6239 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606142807864-6239-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 14:36:10.284224 6239 mini_oidc.cc:152] Starting JWKS server
I20260928 14:36:10.285151 6239 webserver.cc:460] Webserver started at https://127.0.0.1:39097/ using document root <none> and password file <none>
I20260928 14:36:10.285228 6239 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:36:10.285430 6239 webserver.cc:460] Webserver started at http://127.0.0.1:33315/ using document root <none> and password file <none>
2026-09-28T14:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:10Z Disabled control of system clock
I20260928 14:36:10.294400 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:43821
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:43821
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/testchainca.cert with env {}
W20260928 14:36:10.410588 11490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:10.410877 11490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:10.410919 11490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:10.414961 11490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:10.415032 11490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:10.415048 11490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:10.415067 11490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:10.415083 11490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:36:10.415099 11490 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:36:10.415113 11490 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39097/jwks.json
I20260928 14:36:10.420713 11490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:43821
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:43821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:10.421765 11490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:10.422773 11490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:10.431624 11496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:10.431631 11498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:10.431655 11495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:10.431897 11490 server_base.cc:1034] running on GCE node
I20260928 14:36:10.432581 11490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:10.433791 11490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:10.435011 11490 hybrid_clock.cc:648] HybridClock initialized: now 1790606170434974 us; error 50 us; skew 500 ppm
I20260928 14:36:10.437111 11490 webserver.cc:460] Webserver started at http://127.6.23.254:46343/ using document root <none> and password file <none>
I20260928 14:36:10.437721 11490 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:10.437817 11490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:10.438059 11490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:10.440299 11490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "13c40aa2445b43a0ad52d92d051147ee"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "9673e2853b796ca36b5e5abff5822ca9"
server_key_iv: "ce30c8f99c30a02a1d1cade24a9f6236"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.440912 11490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "13c40aa2445b43a0ad52d92d051147ee"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "9673e2853b796ca36b5e5abff5822ca9"
server_key_iv: "ce30c8f99c30a02a1d1cade24a9f6236"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.444808 11490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:36:10.447556 11504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:10.448912 11490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:10.449074 11490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "13c40aa2445b43a0ad52d92d051147ee"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "9673e2853b796ca36b5e5abff5822ca9"
server_key_iv: "ce30c8f99c30a02a1d1cade24a9f6236"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.449203 11490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:10.502650 11490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:10.503584 11490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:10.503810 11490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:10.511086 11490 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:43821
I20260928 14:36:10.511130 11556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:43821 every 8 connection(s)
I20260928 14:36:10.512151 11490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:10.515430 11557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:10.521082 11557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee: Bootstrap starting.
I20260928 14:36:10.521205 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 11490
I20260928 14:36:10.521354 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:10.521639 6239 external_mini_cluster.cc:1428] Setting key bc59c8af1153468941747095dfa80683
I20260928 14:36:10.523958 11557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:10.524878 11557 log.cc:824] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:10.527117 11557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee: No bootstrap required, opened a new log
I20260928 14:36:10.531574 11557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c40aa2445b43a0ad52d92d051147ee" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43821 } }
I20260928 14:36:10.531853 11557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:10.531930 11557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c40aa2445b43a0ad52d92d051147ee, State: Initialized, Role: FOLLOWER
I20260928 14:36:10.532406 11557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c40aa2445b43a0ad52d92d051147ee" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43821 } }
I20260928 14:36:10.532536 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:10.523046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.6.23.254:43821)
0928 14:36:10.523796 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:10.523819 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:10.523864 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:36:10.524775 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:10.524800 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:10.525341 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:10.525669 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:10.526121 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.527575 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.528499 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.529229 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.529383 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.530218 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:10.530227 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:10.530246 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:10.530867 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:10.530938 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:10.530965 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:10.531081 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:10.531203 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:36:10.531503 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":155,"threads_started":1}
I20260928 14:36:10.532579 11557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:10.532636 11557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:10.532733 11557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:10.533738 11557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c40aa2445b43a0ad52d92d051147ee" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43821 } }
I20260928 14:36:10.534102 11557 leader_election.cc:302] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13c40aa2445b43a0ad52d92d051147ee; no voters:
I20260928 14:36:10.534466 11557 leader_election.cc:288] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:10.534598 11562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:10.534833 11562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [term 1 LEADER]: Becoming Leader. State: Replica: 13c40aa2445b43a0ad52d92d051147ee, State: Running, Role: LEADER
I20260928 14:36:10.535164 11562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c40aa2445b43a0ad52d92d051147ee" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43821 } }
I20260928 14:36:10.535677 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:36:10.535909 11557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:10.536897 11563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13c40aa2445b43a0ad52d92d051147ee. Latest consensus state: current_term: 1 leader_uuid: "13c40aa2445b43a0ad52d92d051147ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c40aa2445b43a0ad52d92d051147ee" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43821 } } }
I20260928 14:36:10.537032 11563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:10.537364 11564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13c40aa2445b43a0ad52d92d051147ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13c40aa2445b43a0ad52d92d051147ee" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43821 } } }
I20260928 14:36:10.537460 11564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:10.537768 11571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:10.541025 11571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:10.546907 11571 catalog_manager.cc:1347] Generated new cluster ID: 01687ba9b7c945dd87549258289ac486
I20260928 14:36:10.546995 11571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:10.566885 11571 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:10.567943 11571 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:10.578661 11571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13c40aa2445b43a0ad52d92d051147ee: Generated new TSK 0
I20260928 14:36:10.579510 11571 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:36:10.612480 11587 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 14:36:10.612577 11587 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:36:10.613408 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:10.597108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.6.23.254:43821)
0928 14:36:10.597248 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:10.597255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:10.597473 (+ 218us) server_negotiation.cc:407] Connection header received
0928 14:36:10.597682 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:10.597685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:10.597733 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:10.597853 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:36:10.598095 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.598653 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.599622 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.599848 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.599965 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.600293 (+ 328us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:36:10.612927 (+ 12634us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:36:10.613236 (+ 309us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 14:36:10.613492 11586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:10.597023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43821 (local address 127.0.0.1:41320)
0928 14:36:10.597347 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:10.597360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:10.597527 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:10.597850 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:10.597854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:10.597862 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:36:10.597976 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:10.597983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.598776 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.598780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:10.599499 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:10.599508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.599983 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.599987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:10.600082 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.600105 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:36:10.613137 (+ 13032us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:36:10.613304 (+ 167us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.23.254:43821: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
W20260928 14:36:10.613487 11560 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 14:36:10.614326 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 11490
2026-09-28T14:36:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (369 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 14:36:10.654098 6239 mini_oidc.cc:152] Starting JWKS server
I20260928 14:36:10.655028 6239 webserver.cc:460] Webserver started at https://127.0.0.1:46833/ using document root <none> and password file <none>
I20260928 14:36:10.655160 6239 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:36:10.655468 6239 webserver.cc:460] Webserver started at http://127.0.0.1:37827/ using document root <none> and password file <none>
2026-09-28T14:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:10Z Disabled control of system clock
I20260928 14:36:10.664857 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35971
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:37659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:35971
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46833/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:36:10.783434 11593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:10.783721 11593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:10.783761 11593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:10.788029 11593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:10.788094 11593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:10.788110 11593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:10.788127 11593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:10.788144 11593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:36:10.788160 11593 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:36:10.788174 11593 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46833/jwks.json
I20260928 14:36:10.792985 11593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:37659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:35971
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46833/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:10.794080 11593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:10.795459 11593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:10.804162 11598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:10.804383 11601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:10.804399 11593 server_base.cc:1034] running on GCE node
W20260928 14:36:10.804428 11599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:10.805079 11593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:10.806326 11593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:10.807552 11593 hybrid_clock.cc:648] HybridClock initialized: now 1790606170807537 us; error 28 us; skew 500 ppm
I20260928 14:36:10.809541 11593 webserver.cc:460] Webserver started at http://127.6.23.254:37865/ using document root <none> and password file <none>
I20260928 14:36:10.810134 11593 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:10.810228 11593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:10.810467 11593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:10.812630 11593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "45fc42d341294efa8ea773b0d9ee7e54"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "897bf869cbc893d3d695d4d7ee4d114c"
server_key_iv: "c1ab6092dc302fbe5a4a4e4be80e9725"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.813206 11593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "45fc42d341294efa8ea773b0d9ee7e54"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "897bf869cbc893d3d695d4d7ee4d114c"
server_key_iv: "c1ab6092dc302fbe5a4a4e4be80e9725"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.817138 11593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:10.819640 11607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:10.821033 11593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:10.821188 11593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "45fc42d341294efa8ea773b0d9ee7e54"
format_stamp: "Formatted at 2026-09-28 14:36:10 on dist-test-slave-06lh"
server_key: "897bf869cbc893d3d695d4d7ee4d114c"
server_key_iv: "c1ab6092dc302fbe5a4a4e4be80e9725"
server_key_version: "encryptionkey@0"
I20260928 14:36:10.821313 11593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:10.838915 11593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:10.839726 11593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:10.839941 11593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:10.847136 11593 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:35971
I20260928 14:36:10.847141 11659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:35971 every 8 connection(s)
I20260928 14:36:10.848146 11593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:10.851408 11660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:10.856899 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 11593
I20260928 14:36:10.857036 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:10.857219 11660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54: Bootstrap starting.
I20260928 14:36:10.857342 6239 external_mini_cluster.cc:1428] Setting key a351d243e1e2b9f9fcbffefdc4673b66
I20260928 14:36:10.859895 11660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:10.860662 11660 log.cc:824] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:10.862560 11660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54: No bootstrap required, opened a new log
I20260928 14:36:10.866418 11660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc42d341294efa8ea773b0d9ee7e54" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35971 } }
I20260928 14:36:10.866573 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:10.858650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.6.23.254:35971)
0928 14:36:10.859352 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:10.859367 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:10.859405 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:36:10.860116 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:10.860135 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:10.860548 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:10.860904 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:10.861330 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.862291 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.863270 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.863743 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.863933 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.864811 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:10.864816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:10.864829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:10.865221 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:10.865277 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:10.865296 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:10.865404 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:10.865578 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:36:10.865835 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
I20260928 14:36:10.866678 11660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:10.866761 11660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fc42d341294efa8ea773b0d9ee7e54, State: Initialized, Role: FOLLOWER
I20260928 14:36:10.867274 11660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc42d341294efa8ea773b0d9ee7e54" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35971 } }
I20260928 14:36:10.867449 11660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:10.867506 11660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:10.867600 11660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:10.868579 11660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc42d341294efa8ea773b0d9ee7e54" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35971 } }
I20260928 14:36:10.868947 11660 leader_election.cc:302] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45fc42d341294efa8ea773b0d9ee7e54; no voters:
I20260928 14:36:10.869323 11660 leader_election.cc:288] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:10.869481 11665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:10.869709 11665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [term 1 LEADER]: Becoming Leader. State: Replica: 45fc42d341294efa8ea773b0d9ee7e54, State: Running, Role: LEADER
I20260928 14:36:10.869863 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:36:10.870087 11665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc42d341294efa8ea773b0d9ee7e54" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35971 } }
I20260928 14:36:10.870741 11660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:10.871456 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45fc42d341294efa8ea773b0d9ee7e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc42d341294efa8ea773b0d9ee7e54" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35971 } } }
I20260928 14:36:10.871644 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:10.871914 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45fc42d341294efa8ea773b0d9ee7e54. Latest consensus state: current_term: 1 leader_uuid: "45fc42d341294efa8ea773b0d9ee7e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc42d341294efa8ea773b0d9ee7e54" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35971 } } }
I20260928 14:36:10.872067 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:10.872356 11673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:10.876086 11673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:10.882179 11673 catalog_manager.cc:1347] Generated new cluster ID: ebcff54f8c034a23be0eebba5d7f9f14
I20260928 14:36:10.882261 11673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:10.895148 11673 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:10.896207 11673 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:10.903260 11673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45fc42d341294efa8ea773b0d9ee7e54: Generated new TSK 0
I20260928 14:36:10.904014 11673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:36:10.918567 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:10.913564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.6.23.254:35971)
0928 14:36:10.913703 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:10.913708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:10.913900 (+ 192us) server_negotiation.cc:407] Connection header received
0928 14:36:10.914038 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:10.914041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:10.914088 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:10.914183 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:36:10.914445 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.914987 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.916257 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.916511 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.916591 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.916920 (+ 329us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:36:10.918147 (+ 1227us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:36:10.918409 (+ 262us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 14:36:10.918633 11689 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:10.913489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35971 (local address 127.0.0.1:48502)
0928 14:36:10.913778 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:10.913793 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:10.913921 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:10.914210 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:10.914213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:10.914222 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:36:10.914339 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:10.914346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.915103 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.915106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:10.916120 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:10.916130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:10.916635 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:10.916639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:10.916730 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:10.916758 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:36:10.918315 (+ 1557us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:36:10.918458 (+ 143us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.23.254:35971: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":112,"threads_started":1}
I20260928 14:36:10.919003 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 11593
W20260928 14:36:10.919031 11663 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46833/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
E20260928 14:36:10.931638 11690 webserver.cc:533] Webserver: sslize failed:
E20260928 14:36:10.931725 11690 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T14:36:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (305 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T14:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:10Z Disabled control of system clock
I20260928 14:36:10.969488 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42235
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:46877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:42235
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/testchainca.cert with env {}
W20260928 14:36:11.084277 11694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:11.084657 11694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:11.084722 11694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:11.089118 11694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:11.089202 11694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:11.089218 11694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:11.089236 11694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:11.089251 11694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:36:11.089309 11694 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:36:11.089329 11694 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 14:36:11.095595 11694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:42235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:11.096765 11694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:11.097844 11694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:11.106527 11702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:11.106572 11699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:11.106536 11700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:11.106735 11694 server_base.cc:1034] running on GCE node
I20260928 14:36:11.107443 11694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:11.108608 11694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:11.109774 11694 hybrid_clock.cc:648] HybridClock initialized: now 1790606171109752 us; error 35 us; skew 500 ppm
I20260928 14:36:11.111822 11694 webserver.cc:460] Webserver started at http://127.6.23.254:43913/ using document root <none> and password file <none>
I20260928 14:36:11.112426 11694 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:11.112494 11694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:11.112701 11694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:11.114961 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "3854f35af3174f62900c3f78a7c4dd04"
format_stamp: "Formatted at 2026-09-28 14:36:11 on dist-test-slave-06lh"
server_key: "4f46af7a7ac160c84e26945456ee89f1"
server_key_iv: "04de1d34782f954aa80719fd797bb1ca"
server_key_version: "encryptionkey@0"
I20260928 14:36:11.115622 11694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "3854f35af3174f62900c3f78a7c4dd04"
format_stamp: "Formatted at 2026-09-28 14:36:11 on dist-test-slave-06lh"
server_key: "4f46af7a7ac160c84e26945456ee89f1"
server_key_iv: "04de1d34782f954aa80719fd797bb1ca"
server_key_version: "encryptionkey@0"
I20260928 14:36:11.119462 11694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:36:11.122040 11708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:11.123440 11694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:11.123585 11694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "3854f35af3174f62900c3f78a7c4dd04"
format_stamp: "Formatted at 2026-09-28 14:36:11 on dist-test-slave-06lh"
server_key: "4f46af7a7ac160c84e26945456ee89f1"
server_key_iv: "04de1d34782f954aa80719fd797bb1ca"
server_key_version: "encryptionkey@0"
I20260928 14:36:11.123729 11694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:11.138137 11694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:11.138950 11694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:11.139170 11694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:11.146203 11694 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:42235
I20260928 14:36:11.146231 11760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:42235 every 8 connection(s)
I20260928 14:36:11.147338 11694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:11.150753 11761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:11.155864 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 11694
I20260928 14:36:11.156016 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:11.156317 6239 external_mini_cluster.cc:1428] Setting key 656c855050eb4ae2640cbe7e7cc4a3db
I20260928 14:36:11.157325 11761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04: Bootstrap starting.
I20260928 14:36:11.159938 11761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:11.160804 11761 log.cc:824] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:11.162824 11761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04: No bootstrap required, opened a new log
I20260928 14:36:11.166592 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:11.157849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.6.23.254:42235)
0928 14:36:11.158771 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:11.158786 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:11.158827 (+ 41us) server_negotiation.cc:407] Connection header received
0928 14:36:11.159618 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:11.159639 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:11.160117 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:11.160526 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:11.161021 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:11.162010 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:11.162940 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:11.163507 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:11.163687 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:11.164571 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:11.164576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:11.164589 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:11.164980 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:11.165041 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:11.165061 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:11.165175 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:11.165327 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 14:36:11.165598 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":142,"threads_started":1}
I20260928 14:36:11.166908 11761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3854f35af3174f62900c3f78a7c4dd04" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42235 } }
I20260928 14:36:11.167200 11761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:11.167290 11761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3854f35af3174f62900c3f78a7c4dd04, State: Initialized, Role: FOLLOWER
I20260928 14:36:11.167850 11761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3854f35af3174f62900c3f78a7c4dd04" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42235 } }
I20260928 14:36:11.167979 11761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:11.168088 11761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:11.168207 11761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:11.169390 11761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3854f35af3174f62900c3f78a7c4dd04" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42235 } }
I20260928 14:36:11.169781 11761 leader_election.cc:302] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3854f35af3174f62900c3f78a7c4dd04; no voters:
I20260928 14:36:11.169863 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:36:11.170126 11761 leader_election.cc:288] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:11.170197 11766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:11.170401 11766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [term 1 LEADER]: Becoming Leader. State: Replica: 3854f35af3174f62900c3f78a7c4dd04, State: Running, Role: LEADER
I20260928 14:36:11.170801 11766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3854f35af3174f62900c3f78a7c4dd04" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42235 } }
I20260928 14:36:11.171492 11761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:11.172680 11767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3854f35af3174f62900c3f78a7c4dd04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3854f35af3174f62900c3f78a7c4dd04" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42235 } } }
I20260928 14:36:11.172730 11768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3854f35af3174f62900c3f78a7c4dd04. Latest consensus state: current_term: 1 leader_uuid: "3854f35af3174f62900c3f78a7c4dd04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3854f35af3174f62900c3f78a7c4dd04" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42235 } } }
I20260928 14:36:11.172811 11767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:11.172834 11768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:11.173200 11775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:11.176522 11775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:11.182377 11775 catalog_manager.cc:1347] Generated new cluster ID: 3c8a4cd5969b483b876e44ab2a6ec920
I20260928 14:36:11.182466 11775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:11.194860 11775 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:11.195948 11775 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:11.201212 11775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3854f35af3174f62900c3f78a7c4dd04: Generated new TSK 0
I20260928 14:36:11.202005 11775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:36:11.663837 11764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:11.221413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.6.23.254:42235)
0928 14:36:11.221580 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:11.221586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:11.221736 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:36:11.221914 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:11.221917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:11.221966 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:11.222063 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:36:11.222333 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:11.222892 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:11.224184 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:11.224398 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:11.224482 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:11.224826 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:36:11.663439 (+438613us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:36:11.663715 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51382: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 14:36:11.663951 11764 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51382: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 14:36:11.664119 11790 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:11.221328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42235 (local address 127.0.0.1:51382)
0928 14:36:11.221640 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:11.221657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:11.221780 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:11.222085 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:11.222089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:11.222097 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:36:11.222219 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:11.222225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:11.222999 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:11.223003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:11.224063 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:11.224075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:11.224524 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:11.224528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:11.224629 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:11.224659 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:36:11.663687 (+439028us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:36:11.663901 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.23.254:42235: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":122,"threads_started":1}
I20260928 14:36:11.664791 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 11694
2026-09-28T14:36:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (724 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:11 dist-test-slave-06lh krb5kdc[11802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:11 dist-test-slave-06lh krb5kdc[11802](info): set up 2 sockets
Sep 28 14:36:11 dist-test-slave-06lh krb5kdc[11802](info): commencing operation
krb5kdc: starting...
W20260928 14:36:12.968436 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:13 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:36:13.101717 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45651
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:13.212950 11815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:13.213289 11815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:13.213373 11815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:13.217525 11815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:13.217631 11815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:13.217670 11815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:13.217710 11815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:13.217746 11815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:13.223258 11815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:13.224464 11815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:13.225634 11815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:13.235045 11820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:13.235044 11823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:13.235102 11815 server_base.cc:1034] running on GCE node
W20260928 14:36:13.235038 11821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:13.235950 11815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:13.237215 11815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:13.238435 11815 hybrid_clock.cc:648] HybridClock initialized: now 1790606173238415 us; error 41 us; skew 500 ppm
Sep 28 14:36:13 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:13.241449 11815 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:36:13.242362 11815 webserver.cc:460] Webserver started at http://127.6.23.254:37293/ using document root <none> and password file <none>
I20260928 14:36:13.242956 11815 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:13.243052 11815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:13.243350 11815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:13.245484 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "135fe73a02034099b8a7e64053553058"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "23e3a12fbd35142024233dea9479d9d0"
server_key_iv: "d78e00eab2c812cdb7a08cf4681158e0"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.246097 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "135fe73a02034099b8a7e64053553058"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "23e3a12fbd35142024233dea9479d9d0"
server_key_iv: "d78e00eab2c812cdb7a08cf4681158e0"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.250746 11815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:36:13.253531 11830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:13.254880 11815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:36:13.255051 11815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "135fe73a02034099b8a7e64053553058"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "23e3a12fbd35142024233dea9479d9d0"
server_key_iv: "d78e00eab2c812cdb7a08cf4681158e0"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.255183 11815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:13.270797 11815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:13.274649 11815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:13.274891 11815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:13.282723 11815 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:45651
I20260928 14:36:13.282729 11882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:45651 every 8 connection(s)
I20260928 14:36:13.283818 11815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:13.286895 11883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:13.288161 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 11815
I20260928 14:36:13.288304 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:13.288650 6239 external_mini_cluster.cc:1428] Setting key 09c98b05971f3e0a0e0917c0be53f3fa
I20260928 14:36:13.293975 11883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058: Bootstrap starting.
I20260928 14:36:13.296941 11883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:13.297868 11883 log.cc:824] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:36:13 dist-test-slave-06lh krb5kdc[11802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:13.299999 11883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058: No bootstrap required, opened a new log
I20260928 14:36:13.303589 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:13.290099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.6.23.254:45651)
0928 14:36:13.290534 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:13.290559 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:13.290595 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:36:13.291387 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:13.291407 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:13.291753 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:13.292049 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:13.292978 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.293949 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.295111 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.295666 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.295785 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:13.298669 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:13.298674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:13.298689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:13.298728 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:13.300572 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:13.301024 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:13.301037 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:13.301041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:13.301155 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:13.301459 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:13.301463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:13.301465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:13.301875 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:13.302078 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:13.302403 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 14:36:13.302715 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":148,"threads_started":1}
I20260928 14:36:13.303848 11883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135fe73a02034099b8a7e64053553058" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45651 } }
I20260928 14:36:13.304132 11883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:13.304214 11883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135fe73a02034099b8a7e64053553058, State: Initialized, Role: FOLLOWER
I20260928 14:36:13.304715 11883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135fe73a02034099b8a7e64053553058" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45651 } }
I20260928 14:36:13.304872 11883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:13.304970 11883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:13.305073 11883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:13.306169 11883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135fe73a02034099b8a7e64053553058" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45651 } }
I20260928 14:36:13.306526 11883 leader_election.cc:302] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135fe73a02034099b8a7e64053553058; no voters:
I20260928 14:36:13.306849 11883 leader_election.cc:288] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:13.306993 11888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:13.307278 11888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [term 1 LEADER]: Becoming Leader. State: Replica: 135fe73a02034099b8a7e64053553058, State: Running, Role: LEADER
I20260928 14:36:13.307700 11888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135fe73a02034099b8a7e64053553058" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45651 } }
I20260928 14:36:13.308250 11883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:13.309468 11889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135fe73a02034099b8a7e64053553058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135fe73a02034099b8a7e64053553058" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45651 } } }
I20260928 14:36:13.309599 11889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:13.309998 11897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:13.310838 11890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135fe73a02034099b8a7e64053553058. Latest consensus state: current_term: 1 leader_uuid: "135fe73a02034099b8a7e64053553058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135fe73a02034099b8a7e64053553058" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45651 } } }
I20260928 14:36:13.310950 11890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:13.313395 11897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:13.319272 11897 catalog_manager.cc:1347] Generated new cluster ID: 6cfe3e621f5b412b9001888e44c796f5
I20260928 14:36:13.319353 11897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:13.341126 11897 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:13.342154 11897 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:13.352378 11897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135fe73a02034099b8a7e64053553058: Generated new TSK 0
I20260928 14:36:13.353138 11897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:36:13.384292 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45651
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:13.498291 11909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:13.498565 11909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:13.498611 11909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:13.502813 11909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:13.502875 11909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:13.502966 11909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:13.508397 11909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:13.509548 11909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:13.510672 11909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:13.520190 11917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:13.520190 11914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:13.520386 11909 server_base.cc:1034] running on GCE node
W20260928 14:36:13.520561 11915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:13.521023 11909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:13.521680 11909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:13.522881 11909 hybrid_clock.cc:648] HybridClock initialized: now 1790606173522863 us; error 31 us; skew 500 ppm
Sep 28 14:36:13 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:13.525727 11909 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:36:13.526639 11909 webserver.cc:460] Webserver started at http://127.6.23.193:33047/ using document root <none> and password file <none>
I20260928 14:36:13.527215 11909 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:13.527375 11909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:13.527612 11909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:13.529713 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "b9b6ea6fbde245d6998239efd645523c"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "e46c3eda57c4b71119d8f44c9e4c3b44"
server_key_iv: "46da9bd66c6bc3a57478309e9fc6f459"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.530299 11909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "b9b6ea6fbde245d6998239efd645523c"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "e46c3eda57c4b71119d8f44c9e4c3b44"
server_key_iv: "46da9bd66c6bc3a57478309e9fc6f459"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.534219 11909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 14:36:13.536746 11924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:13.538003 11909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:13.538137 11909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "b9b6ea6fbde245d6998239efd645523c"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "e46c3eda57c4b71119d8f44c9e4c3b44"
server_key_iv: "46da9bd66c6bc3a57478309e9fc6f459"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.538249 11909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:13.549481 11909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:13.552450 11909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:13.552659 11909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:13.553248 11909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:13.554085 11909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:13.554158 11909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:13.554239 11909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:13.554282 11909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:13.563459 11909 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:42715
I20260928 14:36:13.563478 12037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:42715 every 8 connection(s)
I20260928 14:36:13.564414 11909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:13.570901 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 11909
I20260928 14:36:13.571049 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:13.571413 6239 external_mini_cluster.cc:1428] Setting key ce4614f07dee9d3b33f2de66b466116e
Sep 28 14:36:13 dist-test-slave-06lh krb5kdc[11802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:13.578974 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:13.566239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:58207 (local address 127.6.23.254:45651)
0928 14:36:13.566380 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:13.566384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:13.567291 (+ 907us) server_negotiation.cc:407] Connection header received
0928 14:36:13.568697 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:13.568701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:13.568749 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:13.568825 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:13.569995 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.570517 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.571910 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.572159 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.572280 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:13.575087 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:13.575090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:13.575092 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:13.575116 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:13.576637 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:13.577191 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:13.577195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:13.577196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:13.577282 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:13.577711 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:13.577713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:13.577715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:13.577910 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:13.578010 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:13.578722 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 14:36:13.578839 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:36:13.580341 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:13.566659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45651 (local address 127.6.23.193:58207)
0928 14:36:13.567113 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:13.567156 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:13.568498 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:13.568996 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:13.569002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:13.569365 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:13.569866 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:13.569880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.570628 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.570631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:13.571769 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:13.571779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.572339 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.572342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:13.572526 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:13.573163 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:13.573179 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:13.574895 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:13.576775 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:13.576781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:13.576797 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:13.577080 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:13.577386 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:13.577392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:13.577397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:13.577592 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:13.578032 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:13.578041 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:13.578437 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 14:36:13.578695 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 14:36:13.579109 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:36:13.581655 12038 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45651
I20260928 14:36:13.581924 12038 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:13.582460 12038 heartbeater.cc:511] Master 127.6.23.254:45651 requested a full tablet report, sending...
I20260928 14:36:13.584084 11847 ts_manager.cc:194] Registered new tserver with Master: b9b6ea6fbde245d6998239efd645523c (127.6.23.193:42715)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:36:13.585302 11847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:58207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:36:13.602234 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45651
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:13.718997 12043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:13.719288 12043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:13.719333 12043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:13.723541 12043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:13.723613 12043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:13.723727 12043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:13.729133 12043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:13.730273 12043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:13.731400 12043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:13.741457 12049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:13.741465 12051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:13.741472 12048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:13.741801 12043 server_base.cc:1034] running on GCE node
I20260928 14:36:13.742370 12043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:13.743130 12043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:13.744323 12043 hybrid_clock.cc:648] HybridClock initialized: now 1790606173744308 us; error 34 us; skew 500 ppm
Sep 28 14:36:13 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:13.747174 12043 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:36:13.748126 12043 webserver.cc:460] Webserver started at http://127.6.23.194:45325/ using document root <none> and password file <none>
I20260928 14:36:13.748708 12043 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:13.748797 12043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:13.749037 12043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:13.751088 12043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "34a795008c04414688b3438dd93b9608"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "1a92d2fdd5665749f0b831f17ffa86d8"
server_key_iv: "e01548ff0a2274b77a095904c839c5ca"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.751706 12043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "34a795008c04414688b3438dd93b9608"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "1a92d2fdd5665749f0b831f17ffa86d8"
server_key_iv: "e01548ff0a2274b77a095904c839c5ca"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.755478 12043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:36:13.757879 12058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:13.759166 12043 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:13.759356 12043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "34a795008c04414688b3438dd93b9608"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "1a92d2fdd5665749f0b831f17ffa86d8"
server_key_iv: "e01548ff0a2274b77a095904c839c5ca"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.759485 12043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:13.776571 12043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:13.779767 12043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:13.780005 12043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:13.780741 12043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:13.781601 12043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:13.781677 12043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:13.781745 12043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:13.781798 12043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:13.791195 12043 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:43205
I20260928 14:36:13.791236 12171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:43205 every 8 connection(s)
I20260928 14:36:13.792232 12043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:13.799712 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 12043
I20260928 14:36:13.799858 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:13.800174 6239 external_mini_cluster.cc:1428] Setting key 30b8f8d7ff4c7d63da921bdb55d0acf2
Sep 28 14:36:13 dist-test-slave-06lh krb5kdc[11802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:13.806862 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:13.794051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:58591 (local address 127.6.23.254:45651)
0928 14:36:13.794191 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:13.794195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:13.795026 (+ 831us) server_negotiation.cc:407] Connection header received
0928 14:36:13.796452 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:13.796455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:13.796548 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:13.796650 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:13.797817 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.798357 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.799609 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.799904 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.800008 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:13.802695 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:13.802699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:13.802702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:13.802734 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:13.804595 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:13.805108 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:13.805114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:13.805119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:13.805216 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:13.805549 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:13.805555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:13.805560 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:13.805815 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:13.805923 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:13.806512 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 14:36:13.806649 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:36:13.807672 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:13.794399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45651 (local address 127.6.23.194:58591)
0928 14:36:13.794855 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:13.794899 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:13.796235 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:13.796828 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:13.796835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:13.797198 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:13.797665 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:13.797678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.798485 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.798489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:13.799473 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:13.799483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:13.800057 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:13.800061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:13.800260 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:13.800915 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:13.800940 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:13.802510 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:13.804717 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:13.804723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:13.804735 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:13.805004 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:13.805310 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:13.805312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:13.805313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:13.805452 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:13.805917 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:13.805923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:13.806215 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 14:36:13.806421 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 14:36:13.806694 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260928 14:36:13.808939 12172 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45651
I20260928 14:36:13.809224 12172 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:13.809923 12172 heartbeater.cc:511] Master 127.6.23.254:45651 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:36:13.811177 11847 ts_manager.cc:194] Registered new tserver with Master: 34a795008c04414688b3438dd93b9608 (127.6.23.194:43205)
I20260928 14:36:13.811882 11847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:58591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:36:13.831867 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45651
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:13.947044 12177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:13.947428 12177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:13.947480 12177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:13.951731 12177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:13.951802 12177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:13.951861 12177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:13.957345 12177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:13.958495 12177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:13.959692 12177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:13.969780 12185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:13.969801 12182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:13.969810 12183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:13.969998 12177 server_base.cc:1034] running on GCE node
I20260928 14:36:13.970705 12177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:13.971412 12177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:13.972591 12177 hybrid_clock.cc:648] HybridClock initialized: now 1790606173972555 us; error 49 us; skew 500 ppm
Sep 28 14:36:13 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:13.975327 12177 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:36:13.976221 12177 webserver.cc:460] Webserver started at http://127.6.23.195:38695/ using document root <none> and password file <none>
I20260928 14:36:13.976801 12177 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:13.976892 12177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:13.977124 12177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:13.979187 12177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "2f63d17a42754abeb1024f3394648aaa"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "8b746029960ce4e024a62710d86bedbb"
server_key_iv: "6e63720087af051aea2c2ebe6c1d4bf8"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.979806 12177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "2f63d17a42754abeb1024f3394648aaa"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "8b746029960ce4e024a62710d86bedbb"
server_key_iv: "6e63720087af051aea2c2ebe6c1d4bf8"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.983577 12177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:36:13.985942 12192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:13.987375 12177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:13.987521 12177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "2f63d17a42754abeb1024f3394648aaa"
format_stamp: "Formatted at 2026-09-28 14:36:13 on dist-test-slave-06lh"
server_key: "8b746029960ce4e024a62710d86bedbb"
server_key_iv: "6e63720087af051aea2c2ebe6c1d4bf8"
server_key_version: "encryptionkey@0"
I20260928 14:36:13.987648 12177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:14.003388 12177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:14.006575 12177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:14.006806 12177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:14.007526 12177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:14.008450 12177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:14.008520 12177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:14.008594 12177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:14.008647 12177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:14.017743 12177 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:37831
I20260928 14:36:14.017762 12305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:37831 every 8 connection(s)
I20260928 14:36:14.018743 12177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606173, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:14.028968 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 12177
I20260928 14:36:14.029106 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:14.029505 6239 external_mini_cluster.cc:1428] Setting key a15e4a03bc26ceca0e8c0d3af241c791
I20260928 14:36:14.033253 11886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:14.020693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:40819 (local address 127.6.23.254:45651)
0928 14:36:14.020850 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:14.020854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:14.021728 (+ 874us) server_negotiation.cc:407] Connection header received
0928 14:36:14.023114 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:14.023116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:14.023164 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:14.023257 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:14.024600 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.025130 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.026335 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.026568 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.026648 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:14.029239 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:14.029243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:14.029246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:14.029276 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:14.031092 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:14.031652 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:14.031656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:14.031658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:14.031745 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:14.032116 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:14.032119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:14.032121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:14.032319 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:14.032428 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:14.032952 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 14:36:14.033086 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:36:14.034080 12308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:14.021114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45651 (local address 127.6.23.195:40819)
0928 14:36:14.021570 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:14.021614 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:14.022889 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:14.023552 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:14.023559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:14.023922 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:14.024424 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:14.024438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.025273 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.025277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:14.026214 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:14.026224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.026698 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.026701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:14.026858 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:14.027488 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:14.027504 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:14.029016 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:14.031207 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:14.031213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:14.031268 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:14.031544 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:14.031837 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:14.031841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:14.031843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:14.032027 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:14.032428 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:14.032437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:14.032701 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 14:36:14.032889 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 14:36:14.033225 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260928 14:36:14.035180 12306 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45651
I20260928 14:36:14.035480 12306 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:14.035995 12306 heartbeater.cc:511] Master 127.6.23.254:45651 requested a full tablet report, sending...
I20260928 14:36:14.037089 11847 ts_manager.cc:194] Registered new tserver with Master: 2f63d17a42754abeb1024f3394648aaa (127.6.23.195:37831)
I20260928 14:36:14.037667 11847 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:40819
I20260928 14:36:14.043783 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/test-admin.keytab.
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:14.060456 6239 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 14:36:14.063349 12313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:14.061830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.6.23.193:42715)
0928 14:36:14.062127 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:14.062141 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:14.062164 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:36:14.062217 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:14.062223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:14.062440 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:14.062627 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:14.063125 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47958: BlockingRecv error: recv got EOF from 127.0.0.1:47958 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
W20260928 14:36:14.063457 6239 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:14.064561 6239 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:14.065562 6239 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:36:14.076251 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:14.066460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.6.23.193:42715)
0928 14:36:14.066591 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:14.066597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:14.066687 (+ 90us) server_negotiation.cc:407] Connection header received
0928 14:36:14.066852 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:14.066855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:14.066899 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:14.066986 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:14.067650 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.068995 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.069849 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.070245 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.070377 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:14.072928 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:14.072937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:14.072951 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:14.072976 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:14.074520 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:14.074951 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:14.074956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:14.074957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:14.075032 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:14.075318 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:14.075320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:14.075321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:14.075585 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:14.075736 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:14.075909 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 14:36:14.076080 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 14:36:14.080108 12313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:14.079060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.6.23.193:42715)
0928 14:36:14.079188 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:14.079192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:14.079290 (+ 98us) server_negotiation.cc:407] Connection header received
0928 14:36:14.079463 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:14.079466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:14.079513 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:14.079595 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:14.079992 (+ 397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47980: BlockingRecv error: recv got EOF from 127.0.0.1:47980 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:36:14.080284 6239 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:14.081360 6239 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:14.082257 6239 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:36:14.090811 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:14.083072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.6.23.193:42715)
0928 14:36:14.083189 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:14.083192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:14.083343 (+ 151us) server_negotiation.cc:407] Connection header received
0928 14:36:14.083464 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:14.083468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:14.083508 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:14.083597 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:14.084087 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.084613 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.085432 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.085673 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.085776 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:14.087986 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:14.087990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:14.087992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:14.088013 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:14.089207 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:14.089636 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:14.089640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:14.089641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:14.089718 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:14.089998 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:14.090001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:14.090003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:14.090166 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:14.090296 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:14.090537 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 14:36:14.090666 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:36:14.093650 12313 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:14.092805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.6.23.193:42715)
0928 14:36:14.092939 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:14.092943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:14.092981 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:36:14.093082 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:14.093084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:14.093117 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:14.093176 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:14.093533 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48008: BlockingRecv error: recv got EOF from 127.0.0.1:48008 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:36:14.093711 6239 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606142807864-6239-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:14.094702 6239 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:14.095654 6239 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[11802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:36:14.103610 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:14.096411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.6.23.193:42715)
0928 14:36:14.096539 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:14.096545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:14.096646 (+ 101us) server_negotiation.cc:407] Connection header received
0928 14:36:14.096719 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:14.096721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:14.096754 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:14.096813 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:14.097225 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.097728 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.098492 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:14.098701 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:14.098787 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:14.100871 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:14.100874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:14.100876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:14.100896 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:14.102082 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:14.102437 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:14.102441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:14.102442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:14.102521 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:14.102790 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:14.102792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:14.102793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:14.102951 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:14.103044 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:14.103315 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 14:36:14.103437 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:36:14.104667 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 11909
I20260928 14:36:14.110863 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 12043
I20260928 14:36:14.116144 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 12177
I20260928 14:36:14.121594 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 11815
2026-09-28T14:36:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2209 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[12326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[12326](info): set up 2 sockets
Sep 28 14:36:14 dist-test-slave-06lh krb5kdc[12326](info): commencing operation
krb5kdc: starting...
W20260928 14:36:15.159957 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:15 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.254.keytab.
Entry for principal oryx/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.254.keytab.
I20260928 14:36:15.286309 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:43803
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.254.keytab
--principal=oryx/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:43803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:15.397922 12339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:15.398244 12339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:15.398332 12339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:15.402470 12339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:15.402563 12339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:15.402601 12339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:15.402654 12339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:15.402693 12339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:15.408138 12339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:43803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.254.keytab
--principal=oryx/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:43803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:15.409332 12339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:15.410557 12339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:15.419610 12344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:15.419603 12347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:15.419649 12345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:15.419893 12339 server_base.cc:1034] running on GCE node
I20260928 14:36:15.420580 12339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:15.421782 12339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:15.422969 12339 hybrid_clock.cc:648] HybridClock initialized: now 1790606175422952 us; error 36 us; skew 500 ppm
Sep 28 14:36:15 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:15.425760 12339 init.cc:377] Logged in from keytab as oryx/127.6.23.254@KRBTEST.COM (short username oryx)
I20260928 14:36:15.426709 12339 webserver.cc:460] Webserver started at http://127.6.23.254:33489/ using document root <none> and password file <none>
I20260928 14:36:15.427326 12339 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:15.427405 12339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:15.427591 12339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:15.429656 12339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "f65c0fc2dae64ba48a3be443445fd574"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "a0717e3d54c1eb159afee5f1a2cd75d3"
server_key_iv: "f32289b56061987a0a9adbc84906b25a"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.430179 12339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "f65c0fc2dae64ba48a3be443445fd574"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "a0717e3d54c1eb159afee5f1a2cd75d3"
server_key_iv: "f32289b56061987a0a9adbc84906b25a"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.433885 12339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:36:15.436326 12354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:15.437592 12339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:15.437724 12339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "f65c0fc2dae64ba48a3be443445fd574"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "a0717e3d54c1eb159afee5f1a2cd75d3"
server_key_iv: "f32289b56061987a0a9adbc84906b25a"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.437831 12339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:15.452895 12339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:15.455765 12339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:15.455968 12339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:15.463232 12339 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:43803
I20260928 14:36:15.463577 12406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:43803 every 8 connection(s)
I20260928 14:36:15.464359 12339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:15.467338 12407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:15.472679 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 12339
I20260928 14:36:15.472790 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:15.473068 6239 external_mini_cluster.cc:1428] Setting key 8a5b54177eebc13fb0d4cfdb88e75ff9
I20260928 14:36:15.473596 12407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: Bootstrap starting.
I20260928 14:36:15.476341 12407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:15.477183 12407 log.cc:824] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:15.479194 12407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: No bootstrap required, opened a new log
Sep 28 14:36:15 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:15.483474 12407 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } }
I20260928 14:36:15.483714 12407 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:15.483796 12407 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65c0fc2dae64ba48a3be443445fd574, State: Initialized, Role: FOLLOWER
I20260928 14:36:15.484303 12407 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } }
I20260928 14:36:15.484434 12407 raft_consensus.cc:388] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:15.484526 12407 raft_consensus.cc:482] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:15.484634 12407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:15.485638 12407 raft_consensus.cc:504] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } }
I20260928 14:36:15.486009 12407 leader_election.cc:302] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65c0fc2dae64ba48a3be443445fd574; no voters:
I20260928 14:36:15.486344 12407 leader_election.cc:288] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:15.486461 12412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:15.486688 12412 raft_consensus.cc:686] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 1 LEADER]: Becoming Leader. State: Replica: f65c0fc2dae64ba48a3be443445fd574, State: Running, Role: LEADER
I20260928 14:36:15.487033 12412 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } }
I20260928 14:36:15.487807 12407 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:15.488941 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:15.474547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55904 (local address 127.6.23.254:43803)
0928 14:36:15.475025 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:15.475040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:15.475076 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:36:15.475773 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:15.475791 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:15.476142 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:15.476631 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:15.477484 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.478554 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.479521 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.479999 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.480165 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:15.483016 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:15.483021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:15.483034 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:15.483063 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:15.485144 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:15.485617 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:15.485623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:15.485630 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:15.485792 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:15.486091 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:15.486095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:15.486097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:15.486668 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:15.487484 (+ 816us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:15.487634 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 14:36:15.487926 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":163,"threads_started":1}
I20260928 14:36:15.488304 12412 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65c0fc2dae64ba48a3be443445fd574. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } }
I20260928 14:36:15.490393 12412 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:15.490978 12413 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } }
I20260928 14:36:15.491540 12413 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
W20260928 14:36:15.493570 12427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:36:15.493649 12427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:36:15.493737 12428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:15.494555 12428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:15.500168 12428 catalog_manager.cc:1347] Generated new cluster ID: 7ec091b8b0a147b68c5bd8f5351e2e92
I20260928 14:36:15.500253 12428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:15.507332 12428 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:15.508227 12428 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:15.518903 12428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: Generated new TSK 0
I20260928 14:36:15.519781 12428 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab.
Entry for principal oryx/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab.
I20260928 14:36:15.555518 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:43803
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab
--principal=oryx/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:15.672384 12433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:15.672704 12433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:15.672780 12433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:15.676826 12433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:15.676916 12433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:15.677011 12433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:15.682524 12433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab
--principal=oryx/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:15.683650 12433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:15.684719 12433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:36:15.695044 12433 server_base.cc:1034] running on GCE node
W20260928 14:36:15.694981 12438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:15.695009 12441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:15.694981 12439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:15.695706 12433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:15.696430 12433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:15.697621 12433 hybrid_clock.cc:648] HybridClock initialized: now 1790606175697597 us; error 37 us; skew 500 ppm
Sep 28 14:36:15 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:15.700487 12433 init.cc:377] Logged in from keytab as oryx/127.6.23.193@KRBTEST.COM (short username oryx)
I20260928 14:36:15.701359 12433 webserver.cc:460] Webserver started at http://127.6.23.193:42013/ using document root <none> and password file <none>
I20260928 14:36:15.701884 12433 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:15.701941 12433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:15.702104 12433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:15.704232 12433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "0c88a588d8ee4415b59b404a7f1e599e"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "9d618ac94e23f5656cc95f5996d1176d"
server_key_iv: "63ee92a41118d422e9ad1eb697496b9e"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.704757 12433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "0c88a588d8ee4415b59b404a7f1e599e"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "9d618ac94e23f5656cc95f5996d1176d"
server_key_iv: "63ee92a41118d422e9ad1eb697496b9e"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.708550 12433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 14:36:15.711098 12448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:15.712515 12433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:15.712634 12433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "0c88a588d8ee4415b59b404a7f1e599e"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "9d618ac94e23f5656cc95f5996d1176d"
server_key_iv: "63ee92a41118d422e9ad1eb697496b9e"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.712747 12433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:15.729718 12433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:15.733067 12433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:15.733276 12433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:15.733911 12433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:15.734750 12433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:15.734795 12433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:15.734835 12433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:15.734889 12433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:15.744540 12433 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:37771
I20260928 14:36:15.744560 12561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:37771 every 8 connection(s)
I20260928 14:36:15.745472 12433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:15.752712 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 12433
I20260928 14:36:15.752851 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:15.753180 6239 external_mini_cluster.cc:1428] Setting key b74ba0e36409df4f46e37573bcfb3d47
Sep 28 14:36:15 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:15.760315 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:15.747443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:45449 (local address 127.6.23.254:43803)
0928 14:36:15.747591 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:15.747596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:15.748529 (+ 933us) server_negotiation.cc:407] Connection header received
0928 14:36:15.749780 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:15.749783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:15.749829 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:15.749903 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:15.751150 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.751755 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.753293 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.753540 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.753661 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:15.756554 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:15.756557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:15.756559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:15.756584 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:15.758070 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:15.758625 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:15.758629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:15.758631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:15.758715 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:15.759085 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:15.759090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:15.759097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:15.759356 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:15.759479 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:15.760021 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 14:36:15.760149 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:36:15.761195 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:15.747881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:45449)
0928 14:36:15.748365 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:15.748416 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:15.749581 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:15.750092 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:15.750099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:15.750510 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:15.751001 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:15.751015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.751882 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.751885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:15.753154 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:15.753166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.753710 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.753714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:15.753890 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:15.754533 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:15.754549 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:15.756336 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:15.758210 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:15.758216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:15.758234 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:15.758517 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:15.758808 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:15.758812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:15.758814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:15.758982 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:15.759480 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:15.759488 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:15.759758 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 14:36:15.759969 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 14:36:15.760320 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for oryx/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:36:15.762635 12562 heartbeater.cc:347] Connected to a master server at 127.6.23.254:43803
I20260928 14:36:15.762890 12562 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:15.763545 12562 heartbeater.cc:511] Master 127.6.23.254:43803 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.194@KRBTEST.COM" created.
I20260928 14:36:15.765291 12371 ts_manager.cc:194] Registered new tserver with Master: 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771)
I20260928 14:36:15.766939 12371 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:45449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab.
Entry for principal oryx/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab.
I20260928 14:36:15.782021 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:43803
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab
--principal=oryx/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:15.892897 12567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:15.893167 12567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:15.893234 12567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:15.897332 12567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:15.897410 12567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:15.897495 12567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:15.902717 12567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab
--principal=oryx/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:15.903813 12567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:15.904821 12567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:15.914851 12573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:15.914842 12575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:15.914852 12572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:15.915027 12567 server_base.cc:1034] running on GCE node
I20260928 14:36:15.915488 12567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:15.916100 12567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:15.917285 12567 hybrid_clock.cc:648] HybridClock initialized: now 1790606175917251 us; error 50 us; skew 500 ppm
Sep 28 14:36:15 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:15.919966 12567 init.cc:377] Logged in from keytab as oryx/127.6.23.194@KRBTEST.COM (short username oryx)
I20260928 14:36:15.920825 12567 webserver.cc:460] Webserver started at http://127.6.23.194:34633/ using document root <none> and password file <none>
I20260928 14:36:15.921377 12567 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:15.921464 12567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:15.921690 12567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:15.923830 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "0a1bbddd858555556350e24b8c961c94"
server_key_iv: "df4f8a146adcd0544c3b11f520d5a914"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.924396 12567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "0a1bbddd858555556350e24b8c961c94"
server_key_iv: "df4f8a146adcd0544c3b11f520d5a914"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.928087 12567 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:36:15.930397 12582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:15.931617 12567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:15.931739 12567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "0a1bbddd858555556350e24b8c961c94"
server_key_iv: "df4f8a146adcd0544c3b11f520d5a914"
server_key_version: "encryptionkey@0"
I20260928 14:36:15.931843 12567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:15.952944 12567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:15.956339 12567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:15.956548 12567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:15.957158 12567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:15.958011 12567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:15.958079 12567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:15.958153 12567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:15.958199 12567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:15.967319 12567 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:40705
I20260928 14:36:15.967360 12695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:40705 every 8 connection(s)
I20260928 14:36:15.968228 12567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:36:15 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:15.978495 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 12567
I20260928 14:36:15.978634 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:15.979018 6239 external_mini_cluster.cc:1428] Setting key 203197f7afaf7f7f497ac861a6bc36be
I20260928 14:36:15.981871 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:15.970028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:59133 (local address 127.6.23.254:43803)
0928 14:36:15.970178 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:15.970181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:15.970971 (+ 790us) server_negotiation.cc:407] Connection header received
0928 14:36:15.972260 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:15.972264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:15.972323 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:15.972426 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:15.973622 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.974159 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.975444 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.975676 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.975787 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:15.978149 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:15.978152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:15.978154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:15.978176 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:15.979772 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:15.980277 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:15.980291 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:15.980293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:15.980374 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:15.980773 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:15.980776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:15.980777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:15.980953 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:15.981041 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:15.981659 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 14:36:15.981756 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:36:15.982986 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:15.970401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:59133)
0928 14:36:15.970820 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:15.970861 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:15.972047 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:15.972606 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:15.972612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:15.972952 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:15.973474 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:15.973496 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.974287 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.974292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:15.975324 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:15.975334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:15.975824 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:15.975828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:15.976009 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:15.976552 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:15.976568 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:15.977958 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:15.979892 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:15.979897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:15.979912 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:15.980169 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:15.980466 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:15.980471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:15.980473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:15.980670 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:15.981073 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:15.981083 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:15.981400 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 14:36:15.981623 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 14:36:15.982061 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260928 14:36:15.984278 12696 heartbeater.cc:347] Connected to a master server at 127.6.23.254:43803
I20260928 14:36:15.984535 12696 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:15.985062 12696 heartbeater.cc:511] Master 127.6.23.254:43803 requested a full tablet report, sending...
I20260928 14:36:15.986179 12371 ts_manager.cc:194] Registered new tserver with Master: 8f1c4b0dc62f4bae97b75b3cfe7a0f18 (127.6.23.194:40705)
I20260928 14:36:15.987054 12371 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:59133
WARNING: no policy specified for oryx/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab.
Entry for principal oryx/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab.
I20260928 14:36:16.007171 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:43803
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab
--principal=oryx/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:16.118021 12701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:16.118283 12701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:16.118355 12701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:16.122493 12701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:16.122557 12701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:16.122661 12701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:16.128209 12701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab
--principal=oryx/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:43803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:16.129369 12701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:16.130452 12701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:16.141270 12707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:16.141270 12706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:16.141273 12709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:16.141547 12701 server_base.cc:1034] running on GCE node
I20260928 14:36:16.142063 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:16.142690 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:16.143847 12701 hybrid_clock.cc:648] HybridClock initialized: now 1790606176143813 us; error 49 us; skew 500 ppm
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:16.146629 12701 init.cc:377] Logged in from keytab as oryx/127.6.23.195@KRBTEST.COM (short username oryx)
I20260928 14:36:16.147660 12701 webserver.cc:460] Webserver started at http://127.6.23.195:39143/ using document root <none> and password file <none>
I20260928 14:36:16.148245 12701 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:16.148306 12701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:16.148551 12701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:16.150666 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "aaa55d0e51aadc460e4646d3a02ce78a"
server_key_iv: "b8e320ea8760328c4f0c2611d5e15f92"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.151307 12701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "aaa55d0e51aadc460e4646d3a02ce78a"
server_key_iv: "b8e320ea8760328c4f0c2611d5e15f92"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.155053 12701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:16.157516 12716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.158777 12701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:16.158917 12701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "aaa55d0e51aadc460e4646d3a02ce78a"
server_key_iv: "b8e320ea8760328c4f0c2611d5e15f92"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.159044 12701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:16.181607 12701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:16.184729 12701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:16.184939 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:16.185554 12701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:16.186429 12701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:16.186522 12701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.186614 12701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:16.186664 12701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.195917 12701 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:39693
I20260928 14:36:16.195940 12829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:39693 every 8 connection(s)
I20260928 14:36:16.196842 12701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:16.203783 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 12701
I20260928 14:36:16.203922 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:16.204231 6239 external_mini_cluster.cc:1428] Setting key 808f77247b80f66c246c6cf98a06cda0
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:16.211521 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.198790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:37671 (local address 127.6.23.254:43803)
0928 14:36:16.198955 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.198959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.199941 (+ 982us) server_negotiation.cc:407] Connection header received
0928 14:36:16.201209 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.201212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.201263 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.201341 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.202593 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.203130 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.204806 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.205033 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.205162 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.207819 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.207825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.207832 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.207856 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.209347 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.209903 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.209906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.209908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.209986 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.210383 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.210386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.210388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.210578 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.210689 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.211198 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.211354 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:36:16.212364 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.199213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:37671)
0928 14:36:16.199768 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.199817 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.200996 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.201522 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.201530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.201938 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.202449 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.202463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.203273 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.203276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.204667 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.204677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.205200 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.205205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.205405 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.206061 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.206078 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.207641 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.209471 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.209478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.209496 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.209766 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.210095 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.210100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.210102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.210250 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.210686 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.210694 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.210955 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.211148 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.211515 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":118,"threads_started":1}
I20260928 14:36:16.213531 12830 heartbeater.cc:347] Connected to a master server at 127.6.23.254:43803
I20260928 14:36:16.213771 12830 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:16.214247 12830 heartbeater.cc:511] Master 127.6.23.254:43803 requested a full tablet report, sending...
I20260928 14:36:16.215312 12371 ts_manager.cc:194] Registered new tserver with Master: b174366a0a6848f0ac1bfdc4e4b3b7d1 (127.6.23.195:39693)
I20260928 14:36:16.215906 12371 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:37671
I20260928 14:36:16.218344 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.23.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.253.keytab.
Entry for principal oryx/127.6.23.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.253.keytab.
I20260928 14:36:16.246093 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.253:39707
--webserver_interface=127.6.23.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.253.keytab
--principal=oryx/127.6.23.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:43803,127.6.23.253:39707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:16.358145 12835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:16.358426 12835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:16.358466 12835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:16.362633 12835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:16.362697 12835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:16.362717 12835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:16.362735 12835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:16.362749 12835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:16.368202 12835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:43803,127.6.23.253:39707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.253.keytab
--principal=oryx/127.6.23.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.253:39707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.23.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
W20260928 14:36:16.368409 12835 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.23.254:43803,127.6.23.253:39707'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 14:36:16.369248 12835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:16.370221 12835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:16.378726 12841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:16.378744 12840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:16.378751 12843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:16.378854 12835 server_base.cc:1034] running on GCE node
I20260928 14:36:16.379613 12835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:16.380718 12835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:16.381911 12835 hybrid_clock.cc:648] HybridClock initialized: now 1790606176381891 us; error 30 us; skew 500 ppm
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:16.384716 12835 init.cc:377] Logged in from keytab as oryx/127.6.23.253@KRBTEST.COM (short username oryx)
I20260928 14:36:16.385581 12835 webserver.cc:460] Webserver started at http://127.6.23.253:40985/ using document root <none> and password file <none>
I20260928 14:36:16.386149 12835 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:16.386240 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:16.386475 12835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:16.388717 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data/instance:
uuid: "b345d1f0644049e18b10155c09ca8556"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "54ba794bd0920c8aba33e39104e801f6"
server_key_iv: "309b8e071d1114b43a5d0c5da4fe337f"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.389394 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal/instance:
uuid: "b345d1f0644049e18b10155c09ca8556"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "54ba794bd0920c8aba33e39104e801f6"
server_key_iv: "309b8e071d1114b43a5d0c5da4fe337f"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.393263 12835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:16.395912 12850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.397251 12835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:16.397413 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
uuid: "b345d1f0644049e18b10155c09ca8556"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "54ba794bd0920c8aba33e39104e801f6"
server_key_iv: "309b8e071d1114b43a5d0c5da4fe337f"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.397578 12835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:16.410327 12835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:16.413381 12835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:16.413596 12835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:16.420753 12835 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.253:39707
I20260928 14:36:16.421033 12902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.253:39707 every 8 connection(s)
I20260928 14:36:16.421770 12835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
I20260928 14:36:16.422524 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 12835
I20260928 14:36:16.425149 12903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:16.429679 12903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.23.253@KRBTEST.COM
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.253@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:16.438040 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.424213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57338 (local address 127.6.23.253:39707)
0928 14:36:16.424636 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.424650 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.424685 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:36:16.425378 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.425396 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.425717 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.425996 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.426979 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.428078 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.429297 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.429814 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.430084 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.433006 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.433012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.433023 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.433052 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.434980 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.435482 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.435490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.435499 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.435656 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.435930 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.435933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.435934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.436442 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.436688 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.436923 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.437206 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":136,"threads_started":1}
I20260928 14:36:16.441576 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.431899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.6.23.254:43803)
0928 14:36:16.432067 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.432073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.432510 (+ 437us) server_negotiation.cc:407] Connection header received
0928 14:36:16.432568 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.432573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.432636 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.432722 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.433530 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.434260 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.435580 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.435891 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.436008 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.438319 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.438322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.438324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.438345 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.439747 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.440282 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.440286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.440288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.440366 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.440652 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.440657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.440659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.440872 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.440991 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.441293 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.441408 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:36:16.441635 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.431844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:55920)
0928 14:36:16.432248 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.432296 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.432446 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.432770 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.432774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.433241 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.433403 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.433409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.434368 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.434372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.435414 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.435442 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.436016 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.436019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.436122 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.436667 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.436680 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.438112 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.439868 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.439876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.439881 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.440182 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.440474 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.440477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.440478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.440562 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.440980 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.440986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.441099 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.441237 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.441408 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":176,"threads_started":1}
I20260928 14:36:16.442756 12903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.253@KRBTEST.COM for oryx/127.6.23.253@KRBTEST.COM
I20260928 14:36:16.449944 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.443210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.0.0.1:57346)
0928 14:36:16.443340 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.443354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.443427 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.443643 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.443645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.443766 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.443852 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.443856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.444901 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.444903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.445388 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.445395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.445766 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.445769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.445835 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.446269 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.446280 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.447470 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.448844 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.448846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.448847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.449023 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.449245 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.449247 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.449248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.449315 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.449603 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.449605 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.449670 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.449746 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.449835 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 14:36:16.450096 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.443381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.6.23.253:39707)
0928 14:36:16.443483 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.443486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.443496 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:36:16.443524 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.443526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.443556 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.443617 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.443940 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.444802 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.445478 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.445680 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.445751 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.447582 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.447585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.447587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.447605 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.448753 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.449103 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.449106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.449107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.449173 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.449386 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.449388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.449389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.449531 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.449600 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.449838 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.449931 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 14:36:16.452855 12903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Bootstrap starting.
I20260928 14:36:16.455179 12903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:16.455955 12903 log.cc:824] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:16.457819 12903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: No bootstrap required, opened a new log
I20260928 14:36:16.460968 12903 raft_consensus.cc:348] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } }
I20260928 14:36:16.461169 12903 raft_consensus.cc:374] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:16.461254 12903 raft_consensus.cc:729] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b345d1f0644049e18b10155c09ca8556, State: Initialized, Role: FOLLOWER
I20260928 14:36:16.461670 12903 consensus_queue.cc:260] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } }
I20260928 14:36:16.462194 12909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } } }
I20260928 14:36:16.462325 12909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:36:16.463004 12903 sys_catalog.cc:564] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:36:16.467949 12920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:36:16.468035 12920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:36:16.469663 12835 master_runner.cc:428] Detected that this master b345d1f0644049e18b10155c09ca8556 is joining an existing cluster
I20260928 14:36:16.469722 12835 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.23.253:39707
I20260928 14:36:16.481189 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.471809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:55926)
0928 14:36:16.472296 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.472311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.472420 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.472755 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.472757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.472973 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.473098 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.473105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.474196 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.474199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.475147 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.475156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.475856 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.475860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.475947 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.476524 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.476539 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.477030 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.478910 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.478913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.478915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.479765 (+ 850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.480092 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.480099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.480102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.480181 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.480646 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.480649 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.480750 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.480863 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.480995 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260928 14:36:16.481206 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.472184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.6.23.254:43803)
0928 14:36:16.472298 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.472302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.472392 (+ 90us) server_negotiation.cc:407] Connection header received
0928 14:36:16.472528 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.472531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.472576 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.472643 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.473223 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.474053 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.475300 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.475624 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.475695 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.477170 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.477174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.477177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.477200 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.478795 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.479870 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.479876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.479878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.479964 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.480320 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.480323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.480325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.480544 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.480649 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.480938 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.481059 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:36:16.481206 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.472152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.6.23.253:39707)
0928 14:36:16.472260 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.472265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.472665 (+ 400us) server_negotiation.cc:407] Connection header received
0928 14:36:16.472793 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.472796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.472830 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.472873 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.473429 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.474186 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.475337 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.475662 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.475735 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.477392 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.477396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.477398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.477683 (+ 285us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.479571 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.480042 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.480048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.480052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.480134 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.480399 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.480402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.480403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.480577 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.480661 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.480962 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.481058 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:36:16.481547 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.471835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.0.0.1:57362)
0928 14:36:16.472602 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.472613 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.472708 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.472991 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.472994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.473200 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.473316 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.473323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.474316 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.474318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.475194 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.475204 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.475867 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.475870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.476021 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.476775 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.476790 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.477266 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.479679 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.479683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.479685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.479927 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.480228 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.480231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.480233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.480320 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.480701 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.480704 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.480783 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.481251 (+ 468us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.481440 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":54,"thread_start_us":90,"threads_started":1}
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.254@KRBTEST.COM for oryx/127.6.23.253@KRBTEST.COM
I20260928 14:36:16.495119 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.485727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.6.23.253:39707)
0928 14:36:16.485860 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.485865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.486205 (+ 340us) server_negotiation.cc:407] Connection header received
0928 14:36:16.486361 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.486363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.486404 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.486466 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.487194 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.487716 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.488925 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.489234 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.489338 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.491884 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.491891 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.491897 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.491924 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.493382 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.493866 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.493871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.493876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.493963 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.494231 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.494234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.494236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.494406 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.494513 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.494850 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.494965 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:36:16.495150 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.485674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.0.0.1:57366)
0928 14:36:16.486070 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.486123 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.486251 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.486527 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.486532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.486956 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.487088 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.487095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.487814 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.487818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.488821 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.488832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.489364 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.489368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.489489 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.489945 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.489961 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.491723 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.493502 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.493511 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.493517 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.493749 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.494042 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.494046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.494048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.494133 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.494514 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.494522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.494645 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.494788 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.494989 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":107,"threads_started":1}
I20260928 14:36:16.496587 12371 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.23.253:39707: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b345d1f0644049e18b10155c09ca8556"
member_type: NON_VOTER
last_known_addr {
host: "127.6.23.253"
port: 39707
}
attrs {
promote: true
}
}
dest_uuid: "f65c0fc2dae64ba48a3be443445fd574"
cas_config_opid_index: -1
I20260928 14:36:16.497368 12371 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } }
I20260928 14:36:16.499210 12931 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b345d1f0644049e18b10155c09ca8556 (127.6.23.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } }
I20260928 14:36:16.499981 12877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:16.500955 12931 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.23.253:39707
I20260928 14:36:16.501152 12931 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } }
I20260928 14:36:16.501243 12931 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:16.501547 12877 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Refusing update from remote peer f65c0fc2dae64ba48a3be443445fd574: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 14:36:16.502255 12835 master.cc:537] Master@127.6.23.253:39707 shutting down...
I20260928 14:36:16.502297 12931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:36:16.504077 12356 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.6.23.252@KRBTEST.COM; defaulting to no policy
W20260928 14:36:16.507350 12356 consensus_peers.cc:487] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 -> Peer b345d1f0644049e18b10155c09ca8556 (127.6.23.253:39707): Couldn't send request to peer b345d1f0644049e18b10155c09ca8556. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:36:16.507800 12835 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:16.508095 12835 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:16.508214 12835 tablet_replica.cc:330] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.252@KRBTEST.COM" created.
I20260928 14:36:16.513521 12835 master.cc:552] Master@127.6.23.253:39707 shutdown complete.
I20260928 14:36:16.513582 12835 master_runner.cc:305] Clearing existing system tablet
I20260928 14:36:16.513850 12835 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:16.513926 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:16.515527 12835 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:36:16.516538 12936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.516748 12835 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:16.516857 12835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:16.516888 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
uuid: "b345d1f0644049e18b10155c09ca8556"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "54ba794bd0920c8aba33e39104e801f6"
server_key_iv: "309b8e071d1114b43a5d0c5da4fe337f"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.519095 12835 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:16.522543 12835 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:36:16.522641 12835 log.cc:1196] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 14:36:16.522882 12835 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Deleting consensus metadata
I20260928 14:36:16.523077 12835 master_runner.cc:315] Copying system tablet from 127.6.23.254:43803
I20260928 14:36:16.524312 12835 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: tablet copy: Beginning tablet copy session from remote peer at address 127.6.23.254:43803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.252.keytab.
Entry for principal oryx/127.6.23.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.252.keytab.
I20260928 14:36:16.526626 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.252:39359
--webserver_interface=127.6.23.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.252.keytab
--principal=oryx/127.6.23.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:43803,127.6.23.253:39707,127.6.23.252:39359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:36:16.534214 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.524849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:55932)
0928 14:36:16.525128 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.525143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.525261 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.525509 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.525512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.525722 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.525826 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.525833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.526955 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.526958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.527937 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.527947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.528618 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.528622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.528718 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.529756 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.529771 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.530189 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.532474 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.532477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.532479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.532728 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.533058 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.533061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.533063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.533151 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.533642 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.533645 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.533741 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.533861 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.534006 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":116,"threads_started":1}
I20260928 14:36:16.534482 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.525032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.6.23.254:43803)
0928 14:36:16.525169 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.525172 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.525236 (+ 64us) server_negotiation.cc:407] Connection header received
0928 14:36:16.525346 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.525348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.525393 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.525465 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.525971 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.526816 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.528094 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.528492 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.528616 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.530377 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.530384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.530390 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.530426 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.532351 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.532835 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.532839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.532844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.532954 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.533254 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.533260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.533264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.533535 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.533650 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.533971 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.534114 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:36:16.535025 12391 tablet_copy_service.cc:140] P f65c0fc2dae64ba48a3be443445fd574: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b345d1f0644049e18b10155c09ca8556 ({username='oryx', principal='oryx/127.6.23.253@KRBTEST.COM'} at 127.0.0.1:55932)
I20260928 14:36:16.535226 12391 tablet_copy_service.cc:161] P f65c0fc2dae64ba48a3be443445fd574: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b345d1f0644049e18b10155c09ca8556 at {username='oryx', principal='oryx/127.6.23.253@KRBTEST.COM'} at 127.0.0.1:55932: session id = b345d1f0644049e18b10155c09ca8556-00000000000000000000000000000000
I20260928 14:36:16.536862 12391 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:36:16.538693 12835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:36:16.541831 12835 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: tablet copy: Starting download of 0 data blocks...
I20260928 14:36:16.542039 12835 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: tablet copy: Starting download of 1 WAL segments...
I20260928 14:36:16.543852 12835 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:36:16.545723 12391 tablet_copy_service.cc:342] P f65c0fc2dae64ba48a3be443445fd574: Request end of tablet copy session b345d1f0644049e18b10155c09ca8556-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.23.253@KRBTEST.COM'} at 127.0.0.1:55932
I20260928 14:36:16.545866 12391 tablet_copy_service.cc:433] P f65c0fc2dae64ba48a3be443445fd574: ending tablet copy session b345d1f0644049e18b10155c09ca8556-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b345d1f0644049e18b10155c09ca8556
I20260928 14:36:16.554039 12835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:16.555804 12944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:16.556245 12947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:16.557080 12835 server_base.cc:1034] running on GCE node
W20260928 14:36:16.559762 12945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:16.560034 12835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:16.560349 12835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:16.561501 12835 hybrid_clock.cc:648] HybridClock initialized: now 1790606176561475 us; error 43 us; skew 500 ppm
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:16.563150 12835 init.cc:377] Logged in from keytab as oryx/127.6.23.253@KRBTEST.COM (short username oryx)
I20260928 14:36:16.563683 12835 webserver.cc:460] Webserver started at http://127.6.23.253:33851/ using document root <none> and password file <none>
I20260928 14:36:16.563922 12835 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:16.563967 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:16.565303 12835 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:16.566308 12954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.566567 12835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:16.566643 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
uuid: "b345d1f0644049e18b10155c09ca8556"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "54ba794bd0920c8aba33e39104e801f6"
server_key_iv: "309b8e071d1114b43a5d0c5da4fe337f"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.566725 12835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:16.577111 12835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:16.579150 12835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:16.584019 12835 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.253:39707
I20260928 14:36:16.584265 13006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.253:39707 every 8 connection(s)
I20260928 14:36:16.584652 12835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
I20260928 14:36:16.586373 13007 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:36:16.587453 13007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Bootstrap starting.
I20260928 14:36:16.597579 13007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:36:16.598057 13007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: Bootstrap complete.
I20260928 14:36:16.598634 13007 raft_consensus.cc:348] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } }
I20260928 14:36:16.598765 13007 raft_consensus.cc:729] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b345d1f0644049e18b10155c09ca8556, State: Initialized, Role: LEARNER
I20260928 14:36:16.598887 13007 consensus_queue.cc:260] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } }
I20260928 14:36:16.599311 13010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } }
I20260928 14:36:16.599428 13007 sys_catalog.cc:564] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:16.599509 13010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:36:16.602123 13021 catalog_manager.cc:1259] Loaded cluster ID: 7ec091b8b0a147b68c5bd8f5351e2e92
I20260928 14:36:16.602241 13021 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: loading cluster ID for follower catalog manager: success
I20260928 14:36:16.604453 13021 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: acquiring CA information for follower catalog manager: success
I20260928 14:36:16.605825 13021 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 14:36:16.664546 12943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:16.664865 12943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:16.664969 12943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:16.669186 12943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:16.669291 12943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:16.669327 12943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:16.669394 12943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:16.669447 12943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:16.674908 12943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:43803,127.6.23.253:39707,127.6.23.252:39359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.252.keytab
--principal=oryx/127.6.23.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.252:39359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.23.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:16.676257 12943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:16.677345 12943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:16.688537 13029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:16.688585 13026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:16.688727 12943 server_base.cc:1034] running on GCE node
W20260928 14:36:16.688537 13027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:16.689450 12943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:16.690713 12943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:16.691879 12943 hybrid_clock.cc:648] HybridClock initialized: now 1790606176691846 us; error 44 us; skew 500 ppm
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:16.694605 12943 init.cc:377] Logged in from keytab as oryx/127.6.23.252@KRBTEST.COM (short username oryx)
I20260928 14:36:16.695551 12943 webserver.cc:460] Webserver started at http://127.6.23.252:37247/ using document root <none> and password file <none>
I20260928 14:36:16.696123 12943 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:16.696211 12943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:16.696455 12943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:16.698736 12943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data/instance:
uuid: "8c0759c6a8cb49569be2f17b9999c360"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "e9a8f9172f078393e8792ce8d6f5fedf"
server_key_iv: "7e6b928eb549130080c4cf2a33f6df06"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.699579 12943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal/instance:
uuid: "8c0759c6a8cb49569be2f17b9999c360"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "e9a8f9172f078393e8792ce8d6f5fedf"
server_key_iv: "7e6b928eb549130080c4cf2a33f6df06"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.704335 12943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:36:16.706960 13036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.708304 12943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:16.708474 12943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
uuid: "8c0759c6a8cb49569be2f17b9999c360"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "e9a8f9172f078393e8792ce8d6f5fedf"
server_key_iv: "7e6b928eb549130080c4cf2a33f6df06"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.708606 12943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:16.731779 12943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:16.734894 12943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:16.735142 12943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:16.743626 12943 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.252:39359
I20260928 14:36:16.743659 13088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.252:39359 every 8 connection(s)
I20260928 14:36:16.744889 12943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
I20260928 14:36:16.747969 13089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:16.751839 13089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:36:16.753845 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 12943
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.252@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.23.252@KRBTEST.COM
I20260928 14:36:16.766072 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.753981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.6.23.254:43803)
0928 14:36:16.754127 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.754131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.755081 (+ 950us) server_negotiation.cc:407] Connection header received
0928 14:36:16.756188 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.756191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.756257 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.756353 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.757182 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.757731 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.758822 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.759044 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.759166 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.762023 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.762027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.762030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.762060 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.763887 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.764452 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.764455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.764457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.764544 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.764901 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.764905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.764906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.765110 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.765222 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.765733 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.765874 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:36:16.766913 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.754402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:55940)
0928 14:36:16.754915 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.754968 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.755962 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.756403 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.756409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.756770 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.757019 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.757032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.757885 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.757888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.758676 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.758685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.759216 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.759220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.759499 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.760098 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.760114 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.761802 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.764009 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.764019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.764030 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.764341 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.764640 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.764644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.764647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.764793 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.765218 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.765225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.765482 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.765679 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.766023 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":93,"spinlock_wait_cycles":4480,"thread_start_us":155,"threads_started":1}
I20260928 14:36:16.766959 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.755370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.6.23.252:39359)
0928 14:36:16.755624 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.755633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.755665 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:36:16.755978 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.755997 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.756183 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.756369 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.757209 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.758050 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.758891 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.759380 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.759508 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.762518 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.762524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.762535 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.762555 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.764310 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.764735 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.764746 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.764747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.764818 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.765106 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.765109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.765111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.765489 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.765700 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.765939 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.766083 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":171,"spinlock_wait_cycles":1920,"thread_start_us":85,"threads_started":1}
I20260928 14:36:16.768595 13089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:36:16.769524 12562 heartbeater.cc:503] Master 127.6.23.254:43803 was elected leader, sending a full tablet report...
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.252@KRBTEST.COM for oryx/127.6.23.253@KRBTEST.COM
I20260928 14:36:16.776732 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.769134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.0.0.1:57370)
0928 14:36:16.769463 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.769477 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.769559 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.769893 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.769916 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.770093 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.770205 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.770210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.771296 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.771300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.771837 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.771845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.772240 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.772242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.772318 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.772711 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.772721 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.773858 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.775450 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.775452 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.775454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.775636 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.775888 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.775889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.775891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.775958 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.776349 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.776351 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.776423 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.776517 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.776614 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260928 14:36:16.776875 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.769211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.6.23.253:39707)
0928 14:36:16.769497 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.769500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.769513 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:36:16.769709 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.769711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.769771 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.769867 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.770318 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.771158 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.771931 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.772148 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.772225 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.773985 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.773988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.773990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.774013 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.775358 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.775730 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.775732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.775734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.775811 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.776066 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.776069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.776070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.776257 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.776367 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.776579 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.776683 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":101,"threads_started":1}
I20260928 14:36:16.777585 13089 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.252@KRBTEST.COM for oryx/127.6.23.252@KRBTEST.COM
I20260928 14:36:16.785420 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.778054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:39359 (local address 127.0.0.1:35666)
0928 14:36:16.778165 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.778180 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.778254 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.778512 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.778514 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.778637 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.778705 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.778709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.779490 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.779492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.779980 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.779987 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.780958 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.780962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.781017 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.781482 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.781493 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.782637 (+ 1144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.784230 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.784233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.784234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.784407 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.784680 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.784682 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.784683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.784748 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.785068 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.785070 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.785138 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.785214 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.785303 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:36:16.785537 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.778202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.6.23.252:39359)
0928 14:36:16.778329 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.778333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.778345 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:36:16.778375 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.778377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.778409 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.778477 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.778797 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.779379 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.780104 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.780850 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.780950 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.782757 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.782760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.782762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.782782 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.784135 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.784519 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.784521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.784523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.784597 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.784823 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.784825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.784826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.784992 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.785069 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.785270 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.785361 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:36:16.788444 13089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Bootstrap starting.
I20260928 14:36:16.790807 13089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:16.791559 13089 log.cc:824] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:16.793411 13089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: No bootstrap required, opened a new log
I20260928 14:36:16.796489 13089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } }
I20260928 14:36:16.796689 13089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:16.796722 13089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0759c6a8cb49569be2f17b9999c360, State: Initialized, Role: FOLLOWER
I20260928 14:36:16.797156 13089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } }
I20260928 14:36:16.797605 13095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } } }
I20260928 14:36:16.797740 13095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:36:16.798516 13089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:36:16.803755 13106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:36:16.803849 13106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:36:16.807068 12943 master_runner.cc:428] Detected that this master 8c0759c6a8cb49569be2f17b9999c360 is joining an existing cluster
I20260928 14:36:16.807163 12943 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.23.252:39359
I20260928 14:36:16.818727 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.809336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.6.23.254:43803)
0928 14:36:16.809469 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.809474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.809723 (+ 249us) server_negotiation.cc:407] Connection header received
0928 14:36:16.809938 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.809942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.810003 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.810064 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.810747 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.811603 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.812848 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.813195 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.813270 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.814999 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.815003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.815006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.815033 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.816895 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.817396 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.817400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.817403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.817492 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.817791 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.817794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.817795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.818009 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.818082 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.818456 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.818569 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:36:16.818742 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.809242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:55942)
0928 14:36:16.809611 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.809626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.809811 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.810226 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.810228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.810473 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.810621 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.810630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.811734 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.811737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.812719 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.812729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.813427 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.813430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.813645 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.814246 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.814262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.814808 (+ 546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.817019 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.817022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.817024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.817294 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.817568 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.817574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.817576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.817684 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.818159 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.818163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.818275 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.818398 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.818554 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":10,"thread_start_us":129,"threads_started":1}
I20260928 14:36:16.818789 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.809621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.6.23.253:39707)
0928 14:36:16.809773 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.809778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.810227 (+ 449us) server_negotiation.cc:407] Connection header received
0928 14:36:16.810402 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.810407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.810467 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.810521 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.811191 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.811979 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.813784 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.814082 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.814185 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.815421 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.815425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.815427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.815458 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.817040 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.817511 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.817514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.817516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.817599 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.817900 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.817903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.817905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.818116 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.818209 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.818505 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.818633 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:36:16.819295 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.809330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.0.0.1:57382)
0928 14:36:16.810152 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.810167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.810296 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.810657 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.810663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.810946 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.811059 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.811066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.812173 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.812176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.813659 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.813667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.814343 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.814346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.814448 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.814906 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.814925 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.815281 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.817153 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.817158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.817159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.817414 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.817701 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.817704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.817706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.817787 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.818211 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.818214 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.818310 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.818962 (+ 652us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.819097 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260928 14:36:16.821015 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 12433
I20260928 14:36:16.828892 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:37771
--local_ip_for_outbound_sockets=127.6.23.193
--tserver_master_addrs=127.6.23.254:43803
--webserver_port=42013
--webserver_interface=127.6.23.193
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab
--principal=oryx/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707,127.6.23.252:39359 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606175, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.254@KRBTEST.COM for oryx/127.6.23.252@KRBTEST.COM
I20260928 14:36:16.839982 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.809901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:39359 (local address 127.0.0.1:35680)
0928 14:36:16.810237 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.810250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.810374 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.810675 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.810679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.810962 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.811096 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.811104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.812134 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.812137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.813005 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.813015 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.813587 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.813591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.813716 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.835996 (+ 22280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.836015 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.836486 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.838468 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.838471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.838473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.838684 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.838919 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.838921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.838923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.839004 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.839467 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.839470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.839544 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.839632 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.839776 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":26,"spinlock_wait_cycles":2816,"thread_start_us":105,"threads_started":1}
I20260928 14:36:16.840663 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.823481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:39359 (local address 127.0.0.1:35684)
0928 14:36:16.823589 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.823606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.823683 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.824257 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.824260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.824466 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.824570 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.824576 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.825719 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.825722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.826588 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.826597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.827192 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.827195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.827332 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.827864 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.827878 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.829454 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.837975 (+ 8521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.837982 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.837987 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.838311 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.839300 (+ 989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.839306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.839310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.839434 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.839980 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.839985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.840129 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.840229 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.840426 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:36:16.840752 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.809892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.6.23.252:39359)
0928 14:36:16.810001 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.810007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.810310 (+ 303us) server_negotiation.cc:407] Connection header received
0928 14:36:16.810467 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.810469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.810505 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.810550 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.811211 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.812014 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.813124 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.813452 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.813551 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.836658 (+ 23107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.836662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.836665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.836686 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.838356 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.838772 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.838775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.838777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.838838 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.839087 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.839090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.839092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.839351 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.839436 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.840473 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.840582 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":30,"spinlock_wait_cycles":6912}
I20260928 14:36:16.841395 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.823720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35684 (local address 127.6.23.252:39359)
0928 14:36:16.823994 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.824001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.824013 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:36:16.824077 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.824080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.824127 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.824676 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.824776 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.825595 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.826726 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.827073 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.828599 (+ 1526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.829618 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.829622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.829625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.829651 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.837810 (+ 8159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.838477 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.838482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.838487 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.839122 (+ 635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.839575 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.839579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.839583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.839830 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.840878 (+ 1048us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.840967 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.841095 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":101,"threads_started":1}
I20260928 14:36:16.842918 12371 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.23.252:39359: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360"
member_type: NON_VOTER
last_known_addr {
host: "127.6.23.252"
port: 39359
}
attrs {
promote: true
}
}
dest_uuid: "f65c0fc2dae64ba48a3be443445fd574"
cas_config_opid_index: 5
I20260928 14:36:16.843601 12371 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } }
I20260928 14:36:16.844724 12930 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.846190 12930 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.23.252:39359
I20260928 14:36:16.846365 12930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.846454 12930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:16.846903 12943 master.cc:537] Master@127.6.23.252:39359 shutting down...
W20260928 14:36:16.848429 12358 consensus_peers.cc:487] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 -> Peer 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252:39359): Couldn't send request to peer 8c0759c6a8cb49569be2f17b9999c360. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:36:16.851536 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.844922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.0.0.1:57396)
0928 14:36:16.845436 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.845450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.845534 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.846048 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.846051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.846082 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:16.846185 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.846191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.847938 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.847941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.849265 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.849274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.851045 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.851049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.851154 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.851216 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.851314 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.851403 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440}
I20260928 14:36:16.852063 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.845206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57396 (local address 127.6.23.253:39707)
0928 14:36:16.845330 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.845348 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.845613 (+ 265us) server_negotiation.cc:407] Connection header received
0928 14:36:16.845666 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.845669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.845726 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.845810 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:16.846308 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.847133 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.849733 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.850355 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.850466 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.850629 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.851866 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.851949 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:36:16.852627 12981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 LEARNER]: Refusing update from remote peer f65c0fc2dae64ba48a3be443445fd574: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:36:16.853142 12931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 14:36:16.853405 12943 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:16.853722 12943 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:16.853845 12943 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: stopping tablet replica
I20260928 14:36:16.855082 12981 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.857153 13010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65c0fc2dae64ba48a3be443445fd574. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.857260 13010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:36:16.857329 13010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: NON_VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: true } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.857405 13010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:36:16.858007 12931 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: attempting to promote NON_VOTER b345d1f0644049e18b10155c09ca8556 to VOTER
I20260928 14:36:16.858461 12931 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } }
W20260928 14:36:16.860224 12358 consensus_peers.cc:487] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 -> Peer 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252:39359): Couldn't send request to peer 8c0759c6a8cb49569be2f17b9999c360. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:36:16.861006 12980 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 LEARNER]: Refusing update from remote peer f65c0fc2dae64ba48a3be443445fd574: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 14:36:16.861403 12930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 14:36:16.863773 12930 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b345d1f0644049e18b10155c09ca8556 (127.6.23.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.864173 12981 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b345d1f0644049e18b10155c09ca8556 (127.6.23.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.865312 12930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.865419 12930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:16.865531 12930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.865592 12930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:16.866060 13118 sys_catalog.cc:455] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.866250 13118 sys_catalog.cc:458] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:36:16.874711 12943 master.cc:552] Master@127.6.23.252:39359 shutdown complete.
I20260928 14:36:16.874768 12943 master_runner.cc:305] Clearing existing system tablet
I20260928 14:36:16.875020 12943 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:16.875092 12943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:16.876273 12943 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:16.877138 13122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.877347 12943 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:16.877497 12943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:16.877533 12943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
uuid: "8c0759c6a8cb49569be2f17b9999c360"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "e9a8f9172f078393e8792ce8d6f5fedf"
server_key_iv: "7e6b928eb549130080c4cf2a33f6df06"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.879201 12943 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:16.881143 12943 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:36:16.881250 12943 log.cc:1196] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 14:36:16.881497 12943 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Deleting consensus metadata
I20260928 14:36:16.881680 12943 master_runner.cc:315] Copying system tablet from 127.6.23.254:43803
I20260928 14:36:16.882776 12943 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: tablet copy: Beginning tablet copy session from remote peer at address 127.6.23.254:43803
I20260928 14:36:16.891110 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.883777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.6.23.254:43803)
0928 14:36:16.883892 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:16.883899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:16.884042 (+ 143us) server_negotiation.cc:407] Connection header received
0928 14:36:16.884193 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:16.884196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:16.884249 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:16.884343 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:16.884785 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.885318 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.886182 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.886422 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.886531 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.887727 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:16.887730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:16.887733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:16.887758 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:16.889414 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.889836 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.889841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.889846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.889932 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:16.890263 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:16.890266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:16.890268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:16.890451 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:16.890560 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:16.890845 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 14:36:16.890955 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:36:16.891122 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:16.883352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:55950)
0928 14:36:16.883949 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:16.883961 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:16.884096 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:16.884352 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:16.884355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:16.884563 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:16.884661 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.884667 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.885438 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.885443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.886050 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:16.886058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:16.886535 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:16.886539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:16.886639 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:16.887206 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:16.887244 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:16.887581 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:16.889529 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.889534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.889536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.889736 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.890052 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:16.890055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:16.890059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:16.890150 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:16.890549 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:16.890552 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:16.890668 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 14:36:16.890773 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:36:16.890915 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"spinlock_wait_cycles":128896,"thread_start_us":143,"threads_started":1}
I20260928 14:36:16.891942 12391 tablet_copy_service.cc:140] P f65c0fc2dae64ba48a3be443445fd574: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8c0759c6a8cb49569be2f17b9999c360 ({username='oryx', principal='oryx/127.6.23.252@KRBTEST.COM'} at 127.0.0.1:55950)
I20260928 14:36:16.892053 12391 tablet_copy_service.cc:161] P f65c0fc2dae64ba48a3be443445fd574: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8c0759c6a8cb49569be2f17b9999c360 at {username='oryx', principal='oryx/127.6.23.252@KRBTEST.COM'} at 127.0.0.1:55950: session id = 8c0759c6a8cb49569be2f17b9999c360-00000000000000000000000000000000
I20260928 14:36:16.892868 12391 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:36:16.893986 12943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:36:16.895983 12943 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: tablet copy: Starting download of 0 data blocks...
I20260928 14:36:16.896193 12943 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: tablet copy: Starting download of 1 WAL segments...
I20260928 14:36:16.897600 12943 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:36:16.899360 12391 tablet_copy_service.cc:342] P f65c0fc2dae64ba48a3be443445fd574: Request end of tablet copy session 8c0759c6a8cb49569be2f17b9999c360-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.23.252@KRBTEST.COM'} at 127.0.0.1:55950
I20260928 14:36:16.899443 12391 tablet_copy_service.cc:433] P f65c0fc2dae64ba48a3be443445fd574: ending tablet copy session 8c0759c6a8cb49569be2f17b9999c360-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8c0759c6a8cb49569be2f17b9999c360
I20260928 14:36:16.906219 12943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:16.907920 13132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:16.907948 13129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:16.908504 12943 server_base.cc:1034] running on GCE node
W20260928 14:36:16.911445 13130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:16.911731 12943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:16.912057 12943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:16.913192 12943 hybrid_clock.cc:648] HybridClock initialized: now 1790606176913182 us; error 32 us; skew 500 ppm
Sep 28 14:36:16 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606176, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:16.914671 12943 init.cc:377] Logged in from keytab as oryx/127.6.23.252@KRBTEST.COM (short username oryx)
I20260928 14:36:16.915184 12943 webserver.cc:460] Webserver started at http://127.6.23.252:38685/ using document root <none> and password file <none>
I20260928 14:36:16.915439 12943 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:16.915506 12943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:16.916659 12943 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:36:16.917613 13139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:16.917913 12943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:36:16.917994 12943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
uuid: "8c0759c6a8cb49569be2f17b9999c360"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "e9a8f9172f078393e8792ce8d6f5fedf"
server_key_iv: "7e6b928eb549130080c4cf2a33f6df06"
server_key_version: "encryptionkey@0"
I20260928 14:36:16.918085 12943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:16.927407 12943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:16.929381 12943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:16.934440 12943 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.252:39359
I20260928 14:36:16.934670 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.252:39359 every 8 connection(s)
I20260928 14:36:16.935161 12943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
I20260928 14:36:16.936900 13192 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:36:16.937879 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Bootstrap starting.
I20260928 14:36:16.947269 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:36:16.947607 13192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Bootstrap complete.
I20260928 14:36:16.948172 13192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } }
I20260928 14:36:16.948312 13192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c0759c6a8cb49569be2f17b9999c360, State: Initialized, Role: LEARNER
I20260928 14:36:16.948431 13192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } }
I20260928 14:36:16.948776 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:16.948930 13192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:16.948944 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:36:16.952299 13206 catalog_manager.cc:1259] Loaded cluster ID: 7ec091b8b0a147b68c5bd8f5351e2e92
I20260928 14:36:16.952353 13206 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: loading cluster ID for follower catalog manager: success
I20260928 14:36:16.953824 13206 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: acquiring CA information for follower catalog manager: success
I20260928 14:36:16.954914 13206 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:36:16.989329 12696 heartbeater.cc:503] Master 127.6.23.254:43803 was elected leader, sending a full tablet report...
W20260928 14:36:16.993106 13116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:16.993884 13116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:16.993955 13116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:17.000504 13116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:17.000591 13116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:17.000684 13116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:17.008560 13116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab
--principal=oryx/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:37771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=42013
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707,127.6.23.252:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:17.010008 13116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:17.011161 13116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:17.021173 13212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:17.021250 13214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:17.021282 13211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:17.021589 13116 server_base.cc:1034] running on GCE node
I20260928 14:36:17.022033 13116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:17.022676 13116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:17.023859 13116 hybrid_clock.cc:648] HybridClock initialized: now 1790606177023846 us; error 41 us; skew 500 ppm
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:17.037096 13116 init.cc:377] Logged in from keytab as oryx/127.6.23.193@KRBTEST.COM (short username oryx)
I20260928 14:36:17.038193 13116 webserver.cc:460] Webserver started at http://127.6.23.193:42013/ using document root <none> and password file <none>
I20260928 14:36:17.038815 13116 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:17.038928 13116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:17.043211 13116 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:17.045684 13221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.047055 13116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:36:17.047201 13116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "0c88a588d8ee4415b59b404a7f1e599e"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "9d618ac94e23f5656cc95f5996d1176d"
server_key_iv: "63ee92a41118d422e9ad1eb697496b9e"
server_key_version: "encryptionkey@0"
I20260928 14:36:17.047833 13116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:17.074271 13116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:17.077456 13116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:17.077672 13116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:17.078312 13116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:17.079255 13116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:17.079303 13116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.079373 13116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:17.079404 13116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.088606 13116 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:37771
I20260928 14:36:17.088625 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:37771 every 8 connection(s)
I20260928 14:36:17.089748 13116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:17.090440 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 13116
I20260928 14:36:17.090642 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 12567
I20260928 14:36:17.098438 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:40705
--local_ip_for_outbound_sockets=127.6.23.194
--tserver_master_addrs=127.6.23.254:43803
--webserver_port=34633
--webserver_interface=127.6.23.194
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab
--principal=oryx/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707,127.6.23.252:39359 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.252@KRBTEST.COM
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.253@KRBTEST.COM
I20260928 14:36:17.107620 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.094327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:49303 (local address 127.6.23.254:43803)
0928 14:36:17.094702 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.094709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.094726 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:36:17.094777 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.094783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.094840 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.094928 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.096680 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.097559 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.099120 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.099449 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.099573 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.103043 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.103046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.103048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.103074 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.104722 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.105407 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.105412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.105414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.105509 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.105976 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.105980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.105982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.106220 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.106296 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.107110 (+ 814us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.107438 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260928 14:36:17.108693 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.093700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:49303)
0928 14:36:17.094326 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.094339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.094444 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.095077 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.095082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.095606 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.096518 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.096531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.097705 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.097708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.098989 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.098998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.099614 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.099618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.099806 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.100466 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.100484 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.102812 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.104852 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.104860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.104880 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.105281 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.105636 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.105642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.105647 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.105839 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.106395 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.106402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.106799 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.107855 (+ 1056us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.108000 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":297,"spinlock_wait_cycles":896,"thread_start_us":110,"threads_started":1}
I20260928 14:36:17.109472 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.091876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:51401 (local address 127.6.23.253:39707)
0928 14:36:17.092003 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.092010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.093237 (+ 1227us) server_negotiation.cc:407] Connection header received
0928 14:36:17.094483 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.094489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.094544 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.094599 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.096679 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.097503 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.102014 (+ 4511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.102238 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.102324 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.104779 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.104785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.104792 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.104825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.106578 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.107030 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.107036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.107041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.107121 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.107448 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.107452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.107453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.107661 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.107748 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.109182 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.109308 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:36:17.109642 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.094460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:48247 (local address 127.6.23.252:39359)
0928 14:36:17.094794 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.094799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.095125 (+ 326us) server_negotiation.cc:407] Connection header received
0928 14:36:17.095173 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.095179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.095256 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.095335 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.096678 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.097753 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.100300 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.100666 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.100766 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.103125 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.103129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.103132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.103159 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.104978 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.105579 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.105585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.105589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.105679 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.105977 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.105983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.105987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.106233 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.106331 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.108860 (+ 2529us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.109390 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":81,"threads_started":1}
I20260928 14:36:17.108716 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.093861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:39359 (local address 127.6.23.193:48247)
0928 14:36:17.094957 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.094971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.095050 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.095473 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.095476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.095638 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.096506 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.096526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.097865 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.097868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.100159 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.100170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.100881 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.100884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.100973 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.101382 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.101403 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.102958 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.105096 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.105101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.105106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.105405 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.105776 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.105782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.105787 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.105878 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.106395 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.106404 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.106803 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.107032 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.107493 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":1725,"spinlock_wait_cycles":40192,"thread_start_us":108,"threads_started":1}
I20260928 14:36:17.110345 13336 heartbeater.cc:347] Connected to a master server at 127.6.23.254:43803
I20260928 14:36:17.110606 13336 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.110879 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.092164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.6.23.193:51401)
0928 14:36:17.093085 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.093137 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.094232 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.094991 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.095003 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.095765 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.096506 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.096526 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.097675 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.097678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.101041 (+ 3363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.101051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.102342 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.102348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.102457 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.102951 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.102967 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.104627 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.106688 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.106690 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.106692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.106914 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.107220 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.107249 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.107251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.107347 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.108011 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.108014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.108098 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.108186 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.108297 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":2418,"spinlock_wait_cycles":59648,"thread_start_us":579,"threads_started":1}
I20260928 14:36:17.111155 13336 heartbeater.cc:511] Master 127.6.23.254:43803 requested a full tablet report, sending...
I20260928 14:36:17.111423 13335 heartbeater.cc:347] Connected to a master server at 127.6.23.252:39359
I20260928 14:36:17.111518 13335 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.111682 13335 heartbeater.cc:511] Master 127.6.23.252:39359 requested a full tablet report, sending...
I20260928 14:36:17.111987 13337 heartbeater.cc:347] Connected to a master server at 127.6.23.253:39707
I20260928 14:36:17.112072 13337 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.112231 13337 heartbeater.cc:511] Master 127.6.23.253:39707 requested a full tablet report, sending...
I20260928 14:36:17.112603 12371 ts_manager.cc:194] Re-registered known tserver with Master: 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771)
I20260928 14:36:17.112941 13155 ts_manager.cc:194] Registered new tserver with Master: 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771)
I20260928 14:36:17.113226 12971 ts_manager.cc:194] Registered new tserver with Master: 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771)
I20260928 14:36:17.113268 12371 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:49303
I20260928 14:36:17.218181 12830 heartbeater.cc:503] Master 127.6.23.254:43803 was elected leader, sending a full tablet report...
W20260928 14:36:17.225553 13343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:17.225878 13343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:17.225963 13343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:17.230084 13343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:17.230188 13343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:17.230288 13343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:17.235814 13343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab
--principal=oryx/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:40705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=34633
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707,127.6.23.252:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:17.237102 13343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:17.238157 13343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:17.248070 13349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:17.248099 13351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:17.248117 13348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:17.248540 13343 server_base.cc:1034] running on GCE node
I20260928 14:36:17.249027 13343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:17.249776 13343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:17.251019 13343 hybrid_clock.cc:648] HybridClock initialized: now 1790606177250994 us; error 51 us; skew 500 ppm
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:17.253878 13343 init.cc:377] Logged in from keytab as oryx/127.6.23.194@KRBTEST.COM (short username oryx)
I20260928 14:36:17.254774 13343 webserver.cc:460] Webserver started at http://127.6.23.194:34633/ using document root <none> and password file <none>
I20260928 14:36:17.255374 13343 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:17.255458 13343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:17.259332 13343 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:36:17.261338 13358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.262611 13343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:17.262730 13343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18"
format_stamp: "Formatted at 2026-09-28 14:36:15 on dist-test-slave-06lh"
server_key: "0a1bbddd858555556350e24b8c961c94"
server_key_iv: "df4f8a146adcd0544c3b11f520d5a914"
server_key_version: "encryptionkey@0"
I20260928 14:36:17.263326 13343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:17.276710 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.273175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.6.23.252:39359)
0928 14:36:17.273303 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.273308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.273323 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:17.273462 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.273466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.273527 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.273610 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:17.273849 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.274375 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.275800 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.276229 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.276312 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.276425 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.276546 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.276606 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 14:36:17.276729 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.273085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:39359 (local address 127.0.0.1:35690)
0928 14:36:17.273217 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.273233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.273344 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.273635 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.273639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.273655 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:17.273752 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.273757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.274465 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.274468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.275690 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.275701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.276345 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.276348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.276434 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.276457 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.276523 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.276593 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 14:36:17.277576 13166 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 14:36:17.278044 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65c0fc2dae64ba48a3be443445fd574. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:17.278187 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: This master's current role is: LEARNER
I20260928 14:36:17.291679 13343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:17.294680 13343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:17.294893 13343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:17.295619 13343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:17.296525 13343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:17.296572 13343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.296614 13343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:17.296671 13343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.306005 13343 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:40705
I20260928 14:36:17.306025 13471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:40705 every 8 connection(s)
I20260928 14:36:17.307044 13343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:17.310209 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 13343
I20260928 14:36:17.310384 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 12701
I20260928 14:36:17.318267 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:39693
--local_ip_for_outbound_sockets=127.6.23.195
--tserver_master_addrs=127.6.23.254:43803
--webserver_port=39143
--webserver_interface=127.6.23.195
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab
--principal=oryx/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707,127.6.23.252:39359 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.252@KRBTEST.COM
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.253@KRBTEST.COM
I20260928 14:36:17.326395 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.309547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:44325 (local address 127.6.23.253:39707)
0928 14:36:17.309780 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.309784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.310682 (+ 898us) server_negotiation.cc:407] Connection header received
0928 14:36:17.312120 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.312127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.312193 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.312252 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.314713 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.315651 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.317551 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.317903 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.317985 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.322316 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.322320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.322326 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.322367 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.324191 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.324648 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.324652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.324653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.324749 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.325051 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.325056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.325058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.325341 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.325468 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.325990 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.326141 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260928 14:36:17.327570 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.309887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:53047 (local address 127.6.23.252:39359)
0928 14:36:17.311369 (+ 1482us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.311372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.312379 (+ 1007us) server_negotiation.cc:407] Connection header received
0928 14:36:17.312555 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.312558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.312605 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.312655 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.314522 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.315379 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.317574 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.317938 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.318010 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.321597 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.321601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.321604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.321631 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.323568 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.324333 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.324340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.324345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.324446 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.324927 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.324932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.324934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.325167 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.325275 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.327007 (+ 1732us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.327161 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1388}
I20260928 14:36:17.327672 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.309384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:39359 (local address 127.6.23.194:53047)
0928 14:36:17.312300 (+ 2916us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.312313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.312448 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.312804 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.312809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.313572 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.314339 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.314361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.315570 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.315573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.317429 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.317441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.318213 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.318217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.318515 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.319311 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.319338 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.321326 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.323726 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.323734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.323759 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.324192 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.324594 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.324600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.324605 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.324822 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.325285 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.325294 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.325670 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.325925 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.326483 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2827,"mutex_wait_us":1027,"spinlock_wait_cycles":7424,"thread_start_us":105,"threads_started":1}
I20260928 14:36:17.328274 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.309383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.6.23.194:44325)
0928 14:36:17.310524 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.310581 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.311872 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.312554 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.312562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.313884 (+ 1322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.314565 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.314576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.315794 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.315800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.317403 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.317417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.318213 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.318217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.318503 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.319657 (+ 1154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.319674 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.322095 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.324318 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.324321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.324323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.324533 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.324845 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.324850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.324851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.324941 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.325450 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.325452 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.325670 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.325932 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.326428 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":715,"spinlock_wait_cycles":24448,"thread_start_us":137,"threads_started":1}
I20260928 14:36:17.329167 13474 heartbeater.cc:347] Connected to a master server at 127.6.23.253:39707
I20260928 14:36:17.329442 13474 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.329969 13474 heartbeater.cc:511] Master 127.6.23.253:39707 requested a full tablet report, sending...
I20260928 14:36:17.330526 13472 heartbeater.cc:347] Connected to a master server at 127.6.23.252:39359
I20260928 14:36:17.330610 13472 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.330816 13472 heartbeater.cc:511] Master 127.6.23.252:39359 requested a full tablet report, sending...
I20260928 14:36:17.331312 12971 ts_manager.cc:194] Registered new tserver with Master: 8f1c4b0dc62f4bae97b75b3cfe7a0f18 (127.6.23.194:40705)
I20260928 14:36:17.331650 13155 ts_manager.cc:194] Registered new tserver with Master: 8f1c4b0dc62f4bae97b75b3cfe7a0f18 (127.6.23.194:40705)
I20260928 14:36:17.335165 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.309384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:37391)
0928 14:36:17.311769 (+ 2385us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.311780 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.311906 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.312551 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.312560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.313487 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.314339 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.314361 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.315570 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.315573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.317435 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.317441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.318213 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.318217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.318828 (+ 611us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.319496 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.319516 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.331593 (+ 12077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.333762 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.333765 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.333767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.333982 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.334253 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.334255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.334256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.334331 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.334727 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.334729 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.334810 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.334888 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.335022 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2311,"mutex_wait_us":472,"spinlock_wait_cycles":34944,"thread_start_us":148,"threads_started":1}
I20260928 14:36:17.335265 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.309588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:37391 (local address 127.6.23.254:43803)
0928 14:36:17.309725 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.309731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.311875 (+ 2144us) server_negotiation.cc:407] Connection header received
0928 14:36:17.312097 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.312100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.312160 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.312240 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.314524 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.315379 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.317575 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.317928 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.318010 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.331755 (+ 13745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.331762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.331769 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.331806 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.333634 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.334072 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.334075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.334077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.334165 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.334410 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.334413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.334415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.334642 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.334735 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.334960 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.335083 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:36:17.336277 13473 heartbeater.cc:347] Connected to a master server at 127.6.23.254:43803
I20260928 14:36:17.336375 13473 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.336591 13473 heartbeater.cc:511] Master 127.6.23.254:43803 requested a full tablet report, sending...
I20260928 14:36:17.337209 12371 ts_manager.cc:194] Re-registered known tserver with Master: 8f1c4b0dc62f4bae97b75b3cfe7a0f18 (127.6.23.194:40705)
I20260928 14:36:17.337793 12371 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:37391
W20260928 14:36:17.440256 13479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:17.440542 13479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:17.440586 13479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:17.445405 13479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:17.445472 13479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:17.445564 13479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:17.451876 13479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab
--principal=oryx/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:39693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=39143
--tserver_master_addrs=127.6.23.254:43803,127.6.23.253:39707,127.6.23.252:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:17.453083 13479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:17.454160 13479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:17.464352 13487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:17.464362 13485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:17.464623 13479 server_base.cc:1034] running on GCE node
W20260928 14:36:17.465387 13484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:17.465814 13479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:17.466578 13479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:17.467746 13479 hybrid_clock.cc:648] HybridClock initialized: now 1790606177467685 us; error 83 us; skew 500 ppm
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:17.470817 13479 init.cc:377] Logged in from keytab as oryx/127.6.23.195@KRBTEST.COM (short username oryx)
I20260928 14:36:17.471880 13479 webserver.cc:460] Webserver started at http://127.6.23.195:39143/ using document root <none> and password file <none>
I20260928 14:36:17.472491 13479 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:17.472604 13479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:17.476732 13479 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:36:17.478871 13494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.480248 13479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:17.480409 13479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1"
format_stamp: "Formatted at 2026-09-28 14:36:16 on dist-test-slave-06lh"
server_key: "aaa55d0e51aadc460e4646d3a02ce78a"
server_key_iv: "b8e320ea8760328c4f0c2611d5e15f92"
server_key_version: "encryptionkey@0"
I20260928 14:36:17.480997 13479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:17.498976 13479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:17.502242 13479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:17.502446 13479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:17.503062 13479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:17.503981 13479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:17.504052 13479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.504124 13479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:17.504166 13479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:17.513479 13479 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:39693
I20260928 14:36:17.513499 13607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:39693 every 8 connection(s)
I20260928 14:36:17.514549 13479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:17.516891 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 13479
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.252@KRBTEST.COM
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
Sep 28 14:36:17 dist-test-slave-06lh krb5kdc[12326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606177, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.253@KRBTEST.COM
I20260928 14:36:17.531214 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.519568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.6.23.254:43803)
0928 14:36:17.520285 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.520290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.520303 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:36:17.520472 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.520476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.520526 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.520605 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.521631 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.522403 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.523598 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.523988 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.524094 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.526516 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.526524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.526530 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.526563 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.528876 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.529399 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.529403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.529405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.529501 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.529810 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.529813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.529815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.530065 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.530204 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.530824 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.530982 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":79,"threads_started":1}
I20260928 14:36:17.533257 12410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.516364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:34431 (local address 127.6.23.254:43803)
0928 14:36:17.516503 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.516507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.519156 (+ 2649us) server_negotiation.cc:407] Connection header received
0928 14:36:17.519215 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.519218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.519293 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.519361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.521172 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.521971 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.523892 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.524236 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.524311 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.528775 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.528778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.528781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.528806 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.530815 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.531402 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.531409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.531412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.531514 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.531944 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.531948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.531949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.532133 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.532210 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.532941 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.533076 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:36:17.534933 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.516900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:34431)
0928 14:36:17.518825 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.518839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.518926 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.519553 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.519555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.520173 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.520823 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.520851 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.522099 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.522102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.523751 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.523762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.524512 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.524516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.524839 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.526378 (+ 1539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.526391 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.528608 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.530935 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.530941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.530961 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.531284 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.531640 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.531643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.531645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.531837 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.532278 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.532285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.532636 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.533430 (+ 794us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.533823 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1844,"mutex_wait_us":2326,"thread_start_us":100,"threads_started":1}
I20260928 14:36:17.535317 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.519416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.6.23.253:39707)
0928 14:36:17.522937 (+ 3521us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.522941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.522955 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:36:17.522990 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.522993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.523038 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.523120 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.524091 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.524884 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.526135 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.526500 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.526595 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.528351 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.528355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.528360 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.528386 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.532413 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.533115 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.533118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.533120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.533210 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.533761 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.533765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.533767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.533986 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.534118 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.534839 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.534980 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3437,"thread_start_us":72,"threads_started":1}
I20260928 14:36:17.536809 13609 heartbeater.cc:347] Connected to a master server at 127.6.23.254:43803
I20260928 14:36:17.537077 13609 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.537593 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.517101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:44401 (local address 127.6.23.253:39707)
0928 14:36:17.518647 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.518651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.518666 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:17.518844 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.518847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.518904 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.518965 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.520997 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.521832 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.525766 (+ 3934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.526131 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.526218 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.530473 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.530477 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.530480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.530508 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.535850 (+ 5342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.536323 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.536326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.536328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.536412 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.536702 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.536706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.536707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.536918 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.536998 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.537339 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.537450 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1466}
I20260928 14:36:17.537659 13609 heartbeater.cc:511] Master 127.6.23.254:43803 requested a full tablet report, sending...
I20260928 14:36:17.537935 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.518037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:48449 (local address 127.6.23.252:39359)
0928 14:36:17.518517 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.518523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.518542 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:36:17.518789 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.518792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.518867 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.518931 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.521144 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.521905 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.523892 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.524236 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.524311 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.528368 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.528372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.528374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.528404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.530187 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.532551 (+ 2364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.532554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.532556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.532642 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.535602 (+ 2960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.535609 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.535611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.535855 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.535968 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.537358 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.537490 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397}
I20260928 14:36:17.538363 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.516900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.6.23.195:44401)
0928 14:36:17.518250 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.518264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.518561 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.519324 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.519336 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.519983 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.520822 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.520844 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.522011 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.522015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.525441 (+ 3426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.525451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.526986 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.526993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.527101 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.528804 (+ 1703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.528818 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.530312 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.535960 (+ 5648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.535965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.535968 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.536207 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.536524 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.536527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.536529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.536608 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.537053 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.537055 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.537157 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.537759 (+ 602us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.537896 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"mutex_wait_us":1296,"spinlock_wait_cycles":1280,"thread_start_us":98,"threads_started":1}
I20260928 14:36:17.538769 12370 ts_manager.cc:194] Re-registered known tserver with Master: b174366a0a6848f0ac1bfdc4e4b3b7d1 (127.6.23.195:39693)
I20260928 14:36:17.538901 13610 heartbeater.cc:347] Connected to a master server at 127.6.23.253:39707
I20260928 14:36:17.538982 13610 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.538991 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.516900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:39359 (local address 127.6.23.195:48449)
0928 14:36:17.517400 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.517451 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.518543 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.519555 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.519557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.519980 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:17.521023 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.521032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.522027 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.522030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.523751 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.523762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.524512 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.524516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.524841 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.525437 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:17.525467 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:17.528161 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:17.531516 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.531519 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.531522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.531776 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.535222 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:17.535248 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:17.535250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:17.535365 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:17.536958 (+ 1593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:17.536962 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:17.537079 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.538294 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.538664 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":1026,"spinlock_wait_cycles":768,"thread_start_us":138,"threads_started":1}
I20260928 14:36:17.539168 13610 heartbeater.cc:511] Master 127.6.23.253:39707 requested a full tablet report, sending...
I20260928 14:36:17.539559 12370 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:34431
I20260928 14:36:17.539960 13608 heartbeater.cc:347] Connected to a master server at 127.6.23.252:39359
I20260928 14:36:17.540035 13608 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:17.540038 12971 ts_manager.cc:194] Registered new tserver with Master: b174366a0a6848f0ac1bfdc4e4b3b7d1 (127.6.23.195:39693)
I20260928 14:36:17.540196 13608 heartbeater.cc:511] Master 127.6.23.252:39359 requested a full tablet report, sending...
I20260928 14:36:17.540486 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.519949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.6.23.252:39359)
0928 14:36:17.527371 (+ 7422us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.527376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.527391 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:17.527432 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.527436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.527497 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.531992 (+ 4495us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.532108 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.532902 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.534149 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.534521 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.534627 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.536508 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.536512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.536515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.536539 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.538318 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.538812 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.538815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.538817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.538902 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.539213 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.539218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.539221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.539466 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.539591 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.540184 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.540299 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7301,"thread_start_us":86,"threads_started":1}
I20260928 14:36:17.540870 13155 ts_manager.cc:194] Registered new tserver with Master: b174366a0a6848f0ac1bfdc4e4b3b7d1 (127.6.23.195:39693)
I20260928 14:36:17.534596 12371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:36:17.543560 12371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:17.546079 13627 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: NON_VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: true } } }
I20260928 14:36:17.546238 13627 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:17.549636 13626 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: attempting to promote NON_VOTER 8c0759c6a8cb49569be2f17b9999c360 to VOTER
I20260928 14:36:17.550002 13626 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } }
I20260928 14:36:17.551345 12981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Refusing update from remote peer f65c0fc2dae64ba48a3be443445fd574: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 14:36:17.551895 13120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 14:36:17.552067 13165 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 LEARNER]: Refusing update from remote peer f65c0fc2dae64ba48a3be443445fd574: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 14:36:17.553118 13627 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 14:36:17.554270 13120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:17.555051 12981 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:17.555887 13120 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:17.555989 13120 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:17.556141 13627 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:17.556264 13627 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:17.556515 13120 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:17.556612 13120 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:17.555720 13166 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:17.557356 13636 sys_catalog.cc:455] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:17.557492 13636 sys_catalog.cc:458] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:36:17.560412 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f65c0fc2dae64ba48a3be443445fd574" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:17.560539 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:36:17.563444 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.558024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:40705 (local address 127.0.0.1:50868)
0928 14:36:17.558491 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.558508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.558593 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.559323 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.559328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.559349 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:17.559474 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.559481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.561172 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.561175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.562229 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.562238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.563049 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.563052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.563132 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.563154 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.563217 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.563316 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260928 14:36:17.563697 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.558209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.6.23.194:40705)
0928 14:36:17.558600 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.558615 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.558667 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:36:17.558866 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.558872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.559121 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.559645 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:17.559789 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.561042 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.562352 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.562917 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.563051 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.563211 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.563280 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.563371 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":141,"threads_started":1}
I20260928 14:36:17.564721 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.559040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:39693 (local address 127.0.0.1:60044)
0928 14:36:17.559376 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.559389 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.559474 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.560240 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.560243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.560257 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:17.560360 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.560382 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.561993 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.561999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.563174 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.563185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.564306 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.564310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.564407 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.564429 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.564513 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.564586 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260928 14:36:17.564767 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.559443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.6.23.195:39693)
0928 14:36:17.559751 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.559767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.559793 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:36:17.559878 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.559883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.560085 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.560537 (+ 452us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:17.560675 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.561861 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.563363 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.564175 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.564297 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.564432 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.564543 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.564636 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":105,"threads_started":1}
I20260928 14:36:17.566442 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.558709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:37771 (local address 127.0.0.1:55258)
0928 14:36:17.559047 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.559060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.559149 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.561627 (+ 2478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.561631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.561647 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:17.561732 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.561737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.564096 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.564103 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.565165 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.565175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.566073 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.566077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.566163 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.566184 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.566247 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.566317 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
I20260928 14:36:17.566658 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.559683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.6.23.193:37771)
0928 14:36:17.561180 (+ 1497us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.561194 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.561216 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:36:17.561283 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.561287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.561475 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.561863 (+ 388us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:17.562015 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.563933 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.565278 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.565940 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.566072 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.566212 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.566268 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.566366 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1398,"thread_start_us":82,"threads_started":1}
I20260928 14:36:17.567066 13406 tablet_service.cc:1467] Processing CreateTablet for tablet 56531393f0cd4e8f8bd70fae65833dc2 (DEFAULT_TABLE table=test-table [id=90e1bf12947043c6b041e30b82d2e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:17.567493 13542 tablet_service.cc:1467] Processing CreateTablet for tablet 56531393f0cd4e8f8bd70fae65833dc2 (DEFAULT_TABLE table=test-table [id=90e1bf12947043c6b041e30b82d2e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:17.568346 13406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56531393f0cd4e8f8bd70fae65833dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:17.568403 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56531393f0cd4e8f8bd70fae65833dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:17.569344 13269 tablet_service.cc:1467] Processing CreateTablet for tablet 56531393f0cd4e8f8bd70fae65833dc2 (DEFAULT_TABLE table=test-table [id=90e1bf12947043c6b041e30b82d2e7aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:17.570166 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56531393f0cd4e8f8bd70fae65833dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:17.577299 13643 tablet_bootstrap.cc:492] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Bootstrap starting.
I20260928 14:36:17.577458 13642 tablet_bootstrap.cc:492] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Bootstrap starting.
I20260928 14:36:17.579195 13643 tablet_bootstrap.cc:654] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:17.579202 13642 tablet_bootstrap.cc:654] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:17.579638 13644 tablet_bootstrap.cc:492] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Bootstrap starting.
I20260928 14:36:17.580089 13642 log.cc:824] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:17.580107 13643 log.cc:824] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:17.581744 13644 tablet_bootstrap.cc:654] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:17.582370 13642 tablet_bootstrap.cc:492] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: No bootstrap required, opened a new log
I20260928 14:36:17.582589 13642 ts_tablet_manager.cc:1397] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:36:17.582705 13644 log.cc:824] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:17.583819 13643 tablet_bootstrap.cc:492] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: No bootstrap required, opened a new log
I20260928 14:36:17.584038 13643 ts_tablet_manager.cc:1397] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:36:17.584797 13644 tablet_bootstrap.cc:492] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: No bootstrap required, opened a new log
I20260928 14:36:17.584978 13644 ts_tablet_manager.cc:1397] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:36:17.588318 13642 raft_consensus.cc:348] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.588629 13642 raft_consensus.cc:374] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:17.588688 13642 raft_consensus.cc:729] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c88a588d8ee4415b59b404a7f1e599e, State: Initialized, Role: FOLLOWER
I20260928 14:36:17.589293 13642 consensus_queue.cc:260] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.590209 13643 raft_consensus.cc:348] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.590440 13642 ts_tablet_manager.cc:1428] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260928 14:36:17.590492 13643 raft_consensus.cc:374] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:17.590549 13643 raft_consensus.cc:729] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b174366a0a6848f0ac1bfdc4e4b3b7d1, State: Initialized, Role: FOLLOWER
I20260928 14:36:17.590380 13644 raft_consensus.cc:348] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.590628 13644 raft_consensus.cc:374] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:17.590770 13644 raft_consensus.cc:729] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1c4b0dc62f4bae97b75b3cfe7a0f18, State: Initialized, Role: FOLLOWER
I20260928 14:36:17.590848 13336 heartbeater.cc:503] Master 127.6.23.254:43803 was elected leader, sending a full tablet report...
I20260928 14:36:17.591313 13644 consensus_queue.cc:260] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.592303 13644 ts_tablet_manager.cc:1428] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 14:36:17.593565 13473 heartbeater.cc:503] Master 127.6.23.254:43803 was elected leader, sending a full tablet report...
W20260928 14:36:17.594652 13338 tablet.cc:2369] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:17.595217 13643 consensus_queue.cc:260] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.596342 13643 ts_tablet_manager.cc:1428] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260928 14:36:17.598493 13609 heartbeater.cc:503] Master 127.6.23.254:43803 was elected leader, sending a full tablet report...
I20260928 14:36:17.707209 13649 raft_consensus.cc:482] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:17.707588 13649 raft_consensus.cc:504] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.708745 13649 leader_election.cc:288] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771), b174366a0a6848f0ac1bfdc4e4b3b7d1 (127.6.23.195:39693)
I20260928 14:36:17.713918 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.709147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:54817 (local address 127.6.23.193:37771)
0928 14:36:17.709279 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.709286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.709307 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:36:17.709533 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.709537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.709598 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.709685 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:17.710056 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.710770 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.712435 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.713059 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.713196 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.713255 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.713700 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.713782 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:36:17.713918 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.709504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:48719 (local address 127.6.23.195:39693)
0928 14:36:17.709660 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.709667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.709683 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:36:17.709732 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.709735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.709784 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.709880 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:17.710180 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.710811 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.712450 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.713059 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.713196 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.713256 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.713700 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.713779 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:36:17.713927 13476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.709074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:37771 (local address 127.6.23.194:54817)
0928 14:36:17.709227 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.709244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.709364 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.709706 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.709711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.709735 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:17.709917 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.709933 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.710886 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.710892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.712271 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.712278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.713221 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.713225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.713506 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.713568 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.713662 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.713780 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":174}
I20260928 14:36:17.714129 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.709387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:39693 (local address 127.6.23.194:48719)
0928 14:36:17.709553 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:17.709566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:17.709623 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:17.709890 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:17.709893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:17.709911 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:17.710051 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.710058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.710973 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.710976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.712262 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:17.712273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.713221 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.713225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:17.713499 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.713571 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 14:36:17.713673 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:36:17.714020 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":357}
I20260928 14:36:17.714774 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56531393f0cd4e8f8bd70fae65833dc2" candidate_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c88a588d8ee4415b59b404a7f1e599e" is_pre_election: true
I20260928 14:36:17.714778 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56531393f0cd4e8f8bd70fae65833dc2" candidate_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" is_pre_election: true
I20260928 14:36:17.715116 13289 raft_consensus.cc:2454] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1c4b0dc62f4bae97b75b3cfe7a0f18 in term 0.
I20260928 14:36:17.715132 13562 raft_consensus.cc:2454] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1c4b0dc62f4bae97b75b3cfe7a0f18 in term 0.
I20260928 14:36:17.715696 13360 leader_election.cc:302] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c88a588d8ee4415b59b404a7f1e599e, 8f1c4b0dc62f4bae97b75b3cfe7a0f18; no voters:
I20260928 14:36:17.716007 13649 raft_consensus.cc:2781] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:17.716087 13649 raft_consensus.cc:482] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:17.716117 13649 raft_consensus.cc:3037] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:17.717549 13649 raft_consensus.cc:504] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.717903 13649 leader_election.cc:288] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [CANDIDATE]: Term 1 election: Requested vote from peers 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771), b174366a0a6848f0ac1bfdc4e4b3b7d1 (127.6.23.195:39693)
I20260928 14:36:17.718369 13562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56531393f0cd4e8f8bd70fae65833dc2" candidate_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1"
I20260928 14:36:17.718473 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56531393f0cd4e8f8bd70fae65833dc2" candidate_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c88a588d8ee4415b59b404a7f1e599e"
I20260928 14:36:17.718509 13562 raft_consensus.cc:3037] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:17.718621 13289 raft_consensus.cc:3037] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:17.719904 13562 raft_consensus.cc:2454] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1c4b0dc62f4bae97b75b3cfe7a0f18 in term 1.
I20260928 14:36:17.719901 13289 raft_consensus.cc:2454] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1c4b0dc62f4bae97b75b3cfe7a0f18 in term 1.
I20260928 14:36:17.720186 13360 leader_election.cc:302] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1c4b0dc62f4bae97b75b3cfe7a0f18, b174366a0a6848f0ac1bfdc4e4b3b7d1; no voters:
I20260928 14:36:17.720369 13649 raft_consensus.cc:2781] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:17.720630 13649 raft_consensus.cc:686] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 LEADER]: Becoming Leader. State: Replica: 8f1c4b0dc62f4bae97b75b3cfe7a0f18, State: Running, Role: LEADER
I20260928 14:36:17.721025 13649 consensus_queue.cc:237] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } }
I20260928 14:36:17.725584 12371 catalog_manager.cc:5495] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f1c4b0dc62f4bae97b75b3cfe7a0f18 (127.6.23.194). New cstate: current_term: 1 leader_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } health_report { overall_health: UNKNOWN } } }
I20260928 14:36:17.758229 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.754477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.6.23.194:40705)
0928 14:36:17.754611 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.754617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.754632 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:17.754728 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.754732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.754810 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.754937 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:17.755278 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.755827 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.756688 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.756929 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.757040 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.757391 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:17.757834 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:17.757973 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.758016 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.758075 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260928 14:36:17.765955 13611 tablet.cc:2369] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:17.766101 13289 raft_consensus.cc:1261] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 1 FOLLOWER]: Refusing update from remote peer 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:17.766189 13562 raft_consensus.cc:1261] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 FOLLOWER]: Refusing update from remote peer 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:17.767069 13651 consensus_queue.cc:1035] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:17.767345 13649 consensus_queue.cc:1035] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:17.786580 13656 mvcc.cc:205] Tried to move back new op lower bound from 7334322904120438784 to 7334322903948091392. Current Snapshot: MvccSnapshot[applied={T|T < 7334322904120438784 or (T in {7334322904120438784})}]
I20260928 14:36:17.794137 12371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:17.794338 12371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:17.800631 12371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f65c0fc2dae64ba48a3be443445fd574: Sending DeleteTablet for 3 replicas of tablet 56531393f0cd4e8f8bd70fae65833dc2
I20260928 14:36:17.801430 13406 tablet_service.cc:1514] Processing DeleteTablet for tablet 56531393f0cd4e8f8bd70fae65833dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:17 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:50868
I20260928 14:36:17.802026 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet 56531393f0cd4e8f8bd70fae65833dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:17 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:55258
I20260928 14:36:17.802055 13542 tablet_service.cc:1514] Processing DeleteTablet for tablet 56531393f0cd4e8f8bd70fae65833dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:17 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:60044
I20260928 14:36:17.802299 13664 tablet_replica.cc:330] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: stopping tablet replica
I20260928 14:36:17.802697 13664 raft_consensus.cc:2229] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:36:17.804255 13664 raft_consensus.cc:2258] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:17.804448 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 12339
I20260928 14:36:17.805418 13666 tablet_replica.cc:330] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: stopping tablet replica
I20260928 14:36:17.805840 13666 raft_consensus.cc:2229] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:17.806277 13666 raft_consensus.cc:2258] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:17.806761 13664 ts_tablet_manager.cc:1905] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:17.808024 13666 ts_tablet_manager.cc:1905] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:17.809720 13664 ts_tablet_manager.cc:1918] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:17.809826 13664 log.cc:1196] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/56531393f0cd4e8f8bd70fae65833dc2
I20260928 14:36:17.810180 13664 ts_tablet_manager.cc:1939] T 56531393f0cd4e8f8bd70fae65833dc2 P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Deleting consensus metadata
I20260928 14:36:17.811946 13666 ts_tablet_manager.cc:1918] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:17.812127 13666 log.cc:1196] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/56531393f0cd4e8f8bd70fae65833dc2
I20260928 14:36:17.812508 13666 ts_tablet_manager.cc:1939] T 56531393f0cd4e8f8bd70fae65833dc2 P 0c88a588d8ee4415b59b404a7f1e599e: Deleting consensus metadata
W20260928 14:36:17.813458 13360 connection.cc:535] server connection from 127.0.0.1:50868 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 14:36:17.813777 13222 connection.cc:441] server connection from 127.0.0.1:55258 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55258 (request call id 1) could send its response
I20260928 14:36:17.815215 13665 tablet_replica.cc:330] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: stopping tablet replica
I20260928 14:36:17.815629 13665 raft_consensus.cc:2229] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:17.815972 13665 raft_consensus.cc:2258] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:17.817662 13665 ts_tablet_manager.cc:1905] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:17.820880 13665 ts_tablet_manager.cc:1918] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:17.821048 13665 log.cc:1196] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/56531393f0cd4e8f8bd70fae65833dc2
I20260928 14:36:17.821434 13665 ts_tablet_manager.cc:1939] T 56531393f0cd4e8f8bd70fae65833dc2 P b174366a0a6848f0ac1bfdc4e4b3b7d1: Deleting consensus metadata
W20260928 14:36:17.822721 13497 connection.cc:441] server connection from 127.0.0.1:60044 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60044 (request call id 1) could send its response
I20260928 14:36:17.828922 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.819094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35708 (local address 127.6.23.252:39359)
0928 14:36:17.819219 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.819269 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.819513 (+ 244us) server_negotiation.cc:407] Connection header received
0928 14:36:17.819703 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.819707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.819757 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.819808 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.820775 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.821606 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.822813 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.823168 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.823263 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.825266 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.825271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.825274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.825303 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.826981 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.827519 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.827523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.827525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.827607 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.827945 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.827949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.827951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.828165 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.828287 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.828622 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.828741 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:36:17.830521 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.803538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.6.23.253:39707)
0928 14:36:17.803665 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.803669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.819672 (+ 16003us) server_negotiation.cc:407] Connection header received
0928 14:36:17.819880 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.819886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.819947 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.820003 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.820945 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.821775 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.823477 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.823774 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.823865 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.825631 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.825635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.825637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.825664 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.827582 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.828064 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.828067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.828069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.828148 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.828430 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.828436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.828441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.828639 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.830168 (+ 1529us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.830248 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.830358 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:36:17.834139 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.823602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57426 (local address 127.6.23.253:39707)
0928 14:36:17.823729 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.823734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.824595 (+ 861us) server_negotiation.cc:407] Connection header received
0928 14:36:17.824780 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.824784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.824844 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.824902 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.825771 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.826627 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.827914 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.828270 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.828343 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.830441 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.830445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.830448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.830474 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.832458 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.832912 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.832915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.832917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.833002 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.833300 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.833304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.833306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.833485 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.833623 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.833858 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.833977 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:36:17.838241 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:17.804058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35706 (local address 127.6.23.252:39359)
0928 14:36:17.804190 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:17.804194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:17.829887 (+ 25693us) server_negotiation.cc:407] Connection header received
0928 14:36:17.829939 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:17.829943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:17.830000 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:17.830080 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:17.831588 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.832137 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.833162 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:17.833516 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:17.833613 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:17.835187 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:17.835190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:17.835193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:17.835218 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:17.836637 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.837102 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.837105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.837106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.837175 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:17.837475 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:17.837478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:17.837479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:17.837640 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:17.837737 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:17.837991 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 14:36:17.838085 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:36:18.783329 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:18.782790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:39287)
0928 14:36:18.783078 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:18.783181 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
W20260928 14:36:18.783741 13473 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:43803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:18.784969 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:18.784526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:47135)
0928 14:36:18.784764 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:18.784834 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":72,"threads_started":1}
W20260928 14:36:18.785400 13336 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:43803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:18.787124 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:18.786700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:40537)
0928 14:36:18.786935 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:18.787030 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":55,"threads_started":1}
W20260928 14:36:18.787413 13609 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:43803 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:19.299688 13677 raft_consensus.cc:482] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65c0fc2dae64ba48a3be443445fd574)
I20260928 14:36:19.299917 13677 raft_consensus.cc:504] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } }
I20260928 14:36:19.300714 13678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f65c0fc2dae64ba48a3be443445fd574)
I20260928 14:36:19.300894 13678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } }
I20260928 14:36:19.301190 13677 leader_election.cc:288] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803), 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252:39359)
I20260928 14:36:19.301898 13678 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803), b345d1f0644049e18b10155c09ca8556 (127.6.23.253:39707)
I20260928 14:36:19.302338 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.301936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37280)
0928 14:36:19.302165 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.302234 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":57,"threads_started":1}
I20260928 14:36:19.301863 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.301278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37278)
0928 14:36:19.301678 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.301745 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":82,"threads_started":1}
W20260928 14:36:19.302608 13142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
W20260928 14:36:19.302635 12957 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:19.303618 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.303403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37296)
0928 14:36:19.303498 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.303550 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 14:36:19.303966 12957 leader_election.cc:334] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803): Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:19.304057 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.303831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37304)
0928 14:36:19.303946 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.303997 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 14:36:19.304142 13142 leader_election.cc:334] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803): Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:19.306802 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.302066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:39707 (local address 127.0.0.1:52110)
0928 14:36:19.302297 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.302309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:19.302389 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:19.303113 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:19.303117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:19.303137 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:19.303258 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:19.303266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:19.304188 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:19.304191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:19.305375 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:19.305388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:19.306360 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:19.306364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:19.306493 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:19.306523 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:36:19.306596 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:36:19.306696 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260928 14:36:19.306856 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.301702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.6.23.252:39359)
0928 14:36:19.301943 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:19.301950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:19.302241 (+ 291us) server_negotiation.cc:407] Connection header received
0928 14:36:19.302454 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:19.302457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:19.302500 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:19.302552 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:19.303040 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:19.303907 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:19.305514 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:19.306128 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:19.306247 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:19.306308 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:19.306618 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 14:36:19.306735 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260928 14:36:19.307353 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.301438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:39359 (local address 127.0.0.1:54440)
0928 14:36:19.302131 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.302145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:19.302327 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:19.302753 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:19.302756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:19.302773 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:19.302895 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:19.302902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:19.304047 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:19.304050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:19.305339 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:19.305351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:19.306247 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:19.306251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:19.306340 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:19.306364 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:36:19.306675 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 14:36:19.307091 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":79,"threads_started":1}
I20260928 14:36:19.307358 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.302385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.6.23.253:39707)
0928 14:36:19.302792 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:19.302798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:19.302814 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:36:19.302865 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:19.302871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:19.302919 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:19.303005 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:19.303367 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:19.304080 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:19.305514 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:19.306165 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:19.306255 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:19.306308 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:19.307051 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 14:36:19.307133 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":51,"threads_started":1}
I20260928 14:36:19.307801 12981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c0759c6a8cb49569be2f17b9999c360" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b345d1f0644049e18b10155c09ca8556" is_pre_election: true
I20260928 14:36:19.307801 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b345d1f0644049e18b10155c09ca8556" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c0759c6a8cb49569be2f17b9999c360" is_pre_election: true
I20260928 14:36:19.308007 13166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b345d1f0644049e18b10155c09ca8556 in term 1.
I20260928 14:36:19.308009 12981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c0759c6a8cb49569be2f17b9999c360 in term 1.
I20260928 14:36:19.308322 12958 leader_election.cc:302] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c0759c6a8cb49569be2f17b9999c360, b345d1f0644049e18b10155c09ca8556; no voters: f65c0fc2dae64ba48a3be443445fd574
I20260928 14:36:19.308372 13141 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c0759c6a8cb49569be2f17b9999c360, b345d1f0644049e18b10155c09ca8556; no voters: f65c0fc2dae64ba48a3be443445fd574
I20260928 14:36:19.308574 13677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:36:19.308629 13677 raft_consensus.cc:482] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65c0fc2dae64ba48a3be443445fd574)
I20260928 14:36:19.308667 13677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:36:19.308655 13678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:36:19.308741 13678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f65c0fc2dae64ba48a3be443445fd574)
I20260928 14:36:19.308775 13678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:36:19.309671 13677 raft_consensus.cc:504] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } }
I20260928 14:36:19.309680 13678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } }
I20260928 14:36:19.310029 13678 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 2 election: Requested vote from peers f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803), b345d1f0644049e18b10155c09ca8556 (127.6.23.253:39707)
I20260928 14:36:19.310035 13677 leader_election.cc:288] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [CANDIDATE]: Term 2 election: Requested vote from peers f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803), 8c0759c6a8cb49569be2f17b9999c360 (127.6.23.252:39359)
I20260928 14:36:19.310402 12981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c0759c6a8cb49569be2f17b9999c360" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b345d1f0644049e18b10155c09ca8556"
I20260928 14:36:19.310619 12981 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c0759c6a8cb49569be2f17b9999c360 in current term 2: Already voted for candidate b345d1f0644049e18b10155c09ca8556 in this term.
I20260928 14:36:19.310654 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.310381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37324)
0928 14:36:19.310506 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.310586 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:36:19.310827 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b345d1f0644049e18b10155c09ca8556" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c0759c6a8cb49569be2f17b9999c360"
I20260928 14:36:19.310988 13166 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b345d1f0644049e18b10155c09ca8556 in current term 2: Already voted for candidate 8c0759c6a8cb49569be2f17b9999c360 in this term.
I20260928 14:36:19.310905 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.310105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37312)
0928 14:36:19.310659 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.310709 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260928 14:36:19.311309 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.311072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37336)
0928 14:36:19.311158 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.311199 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 14:36:19.311398 13142 leader_election.cc:334] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803): Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:19.311460 13142 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c0759c6a8cb49569be2f17b9999c360; no voters: b345d1f0644049e18b10155c09ca8556, f65c0fc2dae64ba48a3be443445fd574
I20260928 14:36:19.311604 13678 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 14:36:19.311887 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.311684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37338)
0928 14:36:19.311794 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.311837 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 14:36:19.311997 12957 leader_election.cc:334] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803): Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:19.312069 12957 leader_election.cc:302] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b345d1f0644049e18b10155c09ca8556; no voters: 8c0759c6a8cb49569be2f17b9999c360, f65c0fc2dae64ba48a3be443445fd574
I20260928 14:36:19.312192 13677 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 14:36:19.785017 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.784496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:48839)
0928 14:36:19.784799 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.784877 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":97,"threads_started":1}
I20260928 14:36:19.786521 13688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.786054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:55549)
0928 14:36:19.786357 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.786420 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":88,"threads_started":1}
I20260928 14:36:19.788184 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:19.787876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:41719)
0928 14:36:19.788063 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:19.788111 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260928 14:36:20.786559 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:20.786010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:51077)
0928 14:36:20.786373 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:20.786444 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":99,"threads_started":1}
I20260928 14:36:20.787703 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:20.787179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:50561)
0928 14:36:20.787539 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:20.787600 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":85,"threads_started":1}
I20260928 14:36:20.789161 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:20.788845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:48001)
0928 14:36:20.789040 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:20.789090 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":54,"threads_started":1}
I20260928 14:36:21.179518 13693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:21.179661 13693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } }
I20260928 14:36:21.180131 13693 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803), b345d1f0644049e18b10155c09ca8556 (127.6.23.253:39707)
I20260928 14:36:21.180677 12981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c0759c6a8cb49569be2f17b9999c360" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b345d1f0644049e18b10155c09ca8556" is_pre_election: true
I20260928 14:36:21.180748 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.180311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37614)
0928 14:36:21.180599 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.180663 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260928 14:36:21.180815 12981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c0759c6a8cb49569be2f17b9999c360 in term 2.
I20260928 14:36:21.181203 13141 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c0759c6a8cb49569be2f17b9999c360, b345d1f0644049e18b10155c09ca8556; no voters:
I20260928 14:36:21.181411 13693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 14:36:21.181488 13693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:21.181535 13693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:36:21.181797 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.181586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37626)
0928 14:36:21.181691 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.181740 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 14:36:21.181921 13142 leader_election.cc:334] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803): Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:21.182713 13693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } }
I20260928 14:36:21.183051 13693 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 3 election: Requested vote from peers f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803), b345d1f0644049e18b10155c09ca8556 (127.6.23.253:39707)
I20260928 14:36:21.183490 12981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c0759c6a8cb49569be2f17b9999c360" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b345d1f0644049e18b10155c09ca8556"
I20260928 14:36:21.183518 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.183246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37632)
0928 14:36:21.183400 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.183452 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 14:36:21.183637 12981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:36:21.184249 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.184062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37638)
0928 14:36:21.184141 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.184187 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 14:36:21.184358 13142 leader_election.cc:334] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803): Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
I20260928 14:36:21.184783 12981 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c0759c6a8cb49569be2f17b9999c360 in term 3.
I20260928 14:36:21.185079 13141 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c0759c6a8cb49569be2f17b9999c360, b345d1f0644049e18b10155c09ca8556; no voters: f65c0fc2dae64ba48a3be443445fd574
I20260928 14:36:21.185273 13693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:36:21.185480 13693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [term 3 LEADER]: Becoming Leader. State: Replica: 8c0759c6a8cb49569be2f17b9999c360, State: Running, Role: LEADER
I20260928 14:36:21.185766 13693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } }
I20260928 14:36:21.186913 13696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0759c6a8cb49569be2f17b9999c360. Latest consensus state: current_term: 3 leader_uuid: "8c0759c6a8cb49569be2f17b9999c360" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:21.187042 13696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:21.187420 13698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:21.189683 13698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:21.190040 13698 catalog_manager.cc:1259] Loaded cluster ID: 7ec091b8b0a147b68c5bd8f5351e2e92
I20260928 14:36:21.190084 13698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:21.190598 13698 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:21.190940 13698 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Loaded TSK: 0
I20260928 14:36:21.191692 13698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:36:21.209061 13156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:36:21.212280 13156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:21.215340 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.215037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37656)
0928 14:36:21.215157 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.215214 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:36:21.215392 12981 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [term 3 FOLLOWER]: Refusing update from remote peer 8c0759c6a8cb49569be2f17b9999c360: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 14:36:21.215926 13693 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 14:36:21.216130 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.215897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37666)
0928 14:36:21.215999 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.216045 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:36:21.216288 13142 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 -> Peer f65c0fc2dae64ba48a3be443445fd574 (127.6.23.254:43803): Couldn't send request to peer f65c0fc2dae64ba48a3be443445fd574. Status: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:36:21.217465 13700 sys_catalog.cc:455] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0759c6a8cb49569be2f17b9999c360. Latest consensus state: current_term: 3 leader_uuid: "8c0759c6a8cb49569be2f17b9999c360" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:21.217607 13700 sys_catalog.cc:458] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:36:21.218719 13696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8c0759c6a8cb49569be2f17b9999c360" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:21.218868 13696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:21.219152 13700 sys_catalog.cc:455] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8c0759c6a8cb49569be2f17b9999c360" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f65c0fc2dae64ba48a3be443445fd574" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 43803 } } peers { permanent_uuid: "b345d1f0644049e18b10155c09ca8556" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 39707 } attrs { promote: false } } peers { permanent_uuid: "8c0759c6a8cb49569be2f17b9999c360" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 39359 } attrs { promote: false } } }
I20260928 14:36:21.219321 13700 sys_catalog.cc:458] T 00000000000000000000000000000000 P b345d1f0644049e18b10155c09ca8556 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:36:21.220022 13701 mvcc.cc:205] Tried to move back new op lower bound from 7334322918253203456 to 7334322918139650048. Current Snapshot: MvccSnapshot[applied={T|T < 7334322918253203456 or (T in {7334322918253203456})}]
I20260928 14:36:21.229822 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.225336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:40705 (local address 127.0.0.1:41982)
0928 14:36:21.225553 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.225565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:21.225639 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:21.226409 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:21.226412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:21.226432 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:21.226517 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.226523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.227555 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.227561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.228365 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.228376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.229341 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.229345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.229434 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.229501 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 14:36:21.229576 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:36:21.229653 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":6,"thread_start_us":97,"threads_started":1}
I20260928 14:36:21.229997 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.224772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:37771 (local address 127.0.0.1:39680)
0928 14:36:21.224896 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.224914 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:21.225015 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:21.226101 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:21.226104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:21.226121 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:21.226217 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.226223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.227139 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.227144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.228687 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.228695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.229573 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.229576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.229672 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.229699 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:36:21.229763 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:36:21.229816 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":192}
I20260928 14:36:21.230104 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.225360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.6.23.195:39693)
0928 14:36:21.225616 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:21.225623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:21.225638 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:21.225692 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:21.225696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:21.225749 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:21.225845 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:21.226165 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.227025 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.228742 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.229307 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.229431 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.229507 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:21.229880 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 14:36:21.229961 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
I20260928 14:36:21.230261 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.224999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:39693 (local address 127.0.0.1:49356)
0928 14:36:21.225224 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.225237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:21.225323 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:21.225866 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:21.225869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:21.225887 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:21.226031 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.226038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.227155 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.227159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.228610 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.228618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.229432 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.229436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.229597 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.229620 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:36:21.229691 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:36:21.229776 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":161,"thread_start_us":105,"threads_started":1}
I20260928 14:36:21.230427 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.225828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.6.23.194:40705)
0928 14:36:21.226180 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:21.226184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:21.226197 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:36:21.226248 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:21.226251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:21.226296 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:21.226385 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:21.226649 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.227445 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.228502 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.229121 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.229218 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.229284 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:21.230162 (+ 878us) server_negotiation.cc:299] Negotiation successful
0928 14:36:21.230250 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":68,"threads_started":1}
I20260928 14:36:21.230890 13542 tablet_service.cc:1467] Processing CreateTablet for tablet c1c06f38aaff450996fc68cf725fe65d (DEFAULT_TABLE table=test-table [id=ed405857484c4a06886f1bc54b1bb64c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:21.231026 13406 tablet_service.cc:1467] Processing CreateTablet for tablet c1c06f38aaff450996fc68cf725fe65d (DEFAULT_TABLE table=test-table [id=ed405857484c4a06886f1bc54b1bb64c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:21.231184 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.224921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.6.23.193:37771)
0928 14:36:21.225846 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:21.225850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:21.225863 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:36:21.225911 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:21.225914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:21.225970 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:21.226321 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:21.226447 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.227027 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.228814 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.229422 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.229581 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.229642 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:21.229791 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 14:36:21.229882 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":83,"threads_started":1}
I20260928 14:36:21.231249 13542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c06f38aaff450996fc68cf725fe65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:21.231330 13406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c06f38aaff450996fc68cf725fe65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:21.231964 13269 tablet_service.cc:1467] Processing CreateTablet for tablet c1c06f38aaff450996fc68cf725fe65d (DEFAULT_TABLE table=test-table [id=ed405857484c4a06886f1bc54b1bb64c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:21.232214 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c06f38aaff450996fc68cf725fe65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:21.233702 13711 tablet_bootstrap.cc:492] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Bootstrap starting.
I20260928 14:36:21.233870 13712 tablet_bootstrap.cc:492] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: Bootstrap starting.
I20260928 14:36:21.234642 13713 tablet_bootstrap.cc:492] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e: Bootstrap starting.
I20260928 14:36:21.234714 13711 tablet_bootstrap.cc:654] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:21.234714 13712 tablet_bootstrap.cc:654] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:21.235643 13713 tablet_bootstrap.cc:654] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:21.236559 13711 tablet_bootstrap.cc:492] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: No bootstrap required, opened a new log
I20260928 14:36:21.236702 13711 ts_tablet_manager.cc:1397] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 14:36:21.236773 13712 tablet_bootstrap.cc:492] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: No bootstrap required, opened a new log
I20260928 14:36:21.236858 13712 ts_tablet_manager.cc:1397] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:21.236869 13713 tablet_bootstrap.cc:492] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e: No bootstrap required, opened a new log
I20260928 14:36:21.236944 13713 ts_tablet_manager.cc:1397] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:21.237283 13711 raft_consensus.cc:348] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.237283 13712 raft_consensus.cc:348] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.237341 13713 raft_consensus.cc:348] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.237403 13712 raft_consensus.cc:374] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:21.237453 13713 raft_consensus.cc:374] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:21.237453 13712 raft_consensus.cc:729] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b174366a0a6848f0ac1bfdc4e4b3b7d1, State: Initialized, Role: FOLLOWER
I20260928 14:36:21.237498 13713 raft_consensus.cc:729] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c88a588d8ee4415b59b404a7f1e599e, State: Initialized, Role: FOLLOWER
I20260928 14:36:21.237473 13711 raft_consensus.cc:374] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:21.237557 13711 raft_consensus.cc:729] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1c4b0dc62f4bae97b75b3cfe7a0f18, State: Initialized, Role: FOLLOWER
I20260928 14:36:21.237628 13713 consensus_queue.cc:260] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.237632 13712 consensus_queue.cc:260] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.237704 13711 consensus_queue.cc:260] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.238036 13712 ts_tablet_manager.cc:1428] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:21.238057 13713 ts_tablet_manager.cc:1428] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:21.238194 13711 ts_tablet_manager.cc:1428] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:36:21.239207 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.238941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:38269)
0928 14:36:21.239075 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.239137 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 14:36:21.239207 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.238886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:51705)
0928 14:36:21.239062 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.239129 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 14:36:21.241261 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.240957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:43739)
0928 14:36:21.241116 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.241177 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 14:36:21.241439 13472 heartbeater.cc:503] Master 127.6.23.252:39359 was elected leader, sending a full tablet report...
I20260928 14:36:21.241694 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.241376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:39045)
0928 14:36:21.241563 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.241623 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 14:36:21.241900 13335 heartbeater.cc:503] Master 127.6.23.252:39359 was elected leader, sending a full tablet report...
I20260928 14:36:21.243508 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.243264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:37049)
0928 14:36:21.243377 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.243432 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 14:36:21.244150 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.242768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:49297)
0928 14:36:21.244027 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.244079 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1192}
I20260928 14:36:21.244913 13608 heartbeater.cc:503] Master 127.6.23.252:39359 was elected leader, sending a full tablet report...
W20260928 14:36:21.313663 13475 tablet.cc:2369] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:21.611308 13718 raft_consensus.cc:482] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:21.611519 13718 raft_consensus.cc:504] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.612766 13718 leader_election.cc:288] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771), 8f1c4b0dc62f4bae97b75b3cfe7a0f18 (127.6.23.194:40705)
I20260928 14:36:21.617652 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.617252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37676)
0928 14:36:21.617475 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.617551 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 14:36:21.617777 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.612986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:37771 (local address 127.6.23.195:60917)
0928 14:36:21.613201 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.613220 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:21.613329 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:21.613697 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:21.613699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:21.613713 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:21.613840 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.613846 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.614899 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.614902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.616449 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.616461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.617344 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.617348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.617441 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.617492 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:36:21.617573 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:36:21.617645 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":106}
I20260928 14:36:21.617844 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.613105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:60917 (local address 127.6.23.193:37771)
0928 14:36:21.613252 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:21.613256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:21.613277 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:36:21.613479 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:21.613482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:21.613570 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:21.613653 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:21.613982 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.614779 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.616585 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.617215 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.617368 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.617423 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:21.617618 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 14:36:21.617705 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:36:21.617982 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.613151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:51919 (local address 127.6.23.194:40705)
0928 14:36:21.613289 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:21.613293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:21.613564 (+ 271us) server_negotiation.cc:407] Connection header received
0928 14:36:21.613646 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:21.613649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:21.613700 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:21.613775 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:21.613980 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.614778 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.616585 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.617220 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.617384 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.617427 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:21.617785 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 14:36:21.617865 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:36:21.618259 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c06f38aaff450996fc68cf725fe65d" candidate_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c88a588d8ee4415b59b404a7f1e599e" is_pre_election: true
I20260928 14:36:21.618372 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.613049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:40705 (local address 127.6.23.195:51919)
0928 14:36:21.613465 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.613476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:21.613567 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:21.613796 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:21.613799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:21.613809 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:21.613869 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.613877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.614899 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.614902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.616449 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:21.616461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.617513 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.617516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:21.617644 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.617666 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:36:21.618171 (+ 505us) client_negotiation.cc:240] Negotiation successful
0928 14:36:21.618244 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":123,"thread_start_us":128,"threads_started":1}
I20260928 14:36:21.618412 13289 raft_consensus.cc:2454] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b174366a0a6848f0ac1bfdc4e4b3b7d1 in term 0.
I20260928 14:36:21.618523 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.618279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.0.0.1:37678)
0928 14:36:21.618408 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.618462 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 14:36:21.618894 13496 leader_election.cc:302] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c88a588d8ee4415b59b404a7f1e599e, b174366a0a6848f0ac1bfdc4e4b3b7d1; no voters:
I20260928 14:36:21.619045 13426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c06f38aaff450996fc68cf725fe65d" candidate_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" is_pre_election: true
I20260928 14:36:21.619211 13718 raft_consensus.cc:2781] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:21.619320 13718 raft_consensus.cc:482] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:21.619326 13426 raft_consensus.cc:2454] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b174366a0a6848f0ac1bfdc4e4b3b7d1 in term 0.
I20260928 14:36:21.619378 13718 raft_consensus.cc:3037] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:21.620436 13718 raft_consensus.cc:504] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.620929 13718 leader_election.cc:288] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [CANDIDATE]: Term 1 election: Requested vote from peers 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771), 8f1c4b0dc62f4bae97b75b3cfe7a0f18 (127.6.23.194:40705)
I20260928 14:36:21.621309 13289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c06f38aaff450996fc68cf725fe65d" candidate_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c88a588d8ee4415b59b404a7f1e599e"
I20260928 14:36:21.621309 13426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c06f38aaff450996fc68cf725fe65d" candidate_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18"
I20260928 14:36:21.621434 13426 raft_consensus.cc:3037] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:21.621434 13289 raft_consensus.cc:3037] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:21.622556 13426 raft_consensus.cc:2454] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b174366a0a6848f0ac1bfdc4e4b3b7d1 in term 1.
I20260928 14:36:21.622556 13289 raft_consensus.cc:2454] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b174366a0a6848f0ac1bfdc4e4b3b7d1 in term 1.
I20260928 14:36:21.622834 13498 leader_election.cc:302] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1c4b0dc62f4bae97b75b3cfe7a0f18, b174366a0a6848f0ac1bfdc4e4b3b7d1; no voters:
I20260928 14:36:21.623036 13718 raft_consensus.cc:2781] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:21.623323 13718 raft_consensus.cc:686] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 LEADER]: Becoming Leader. State: Replica: b174366a0a6848f0ac1bfdc4e4b3b7d1, State: Running, Role: LEADER
I20260928 14:36:21.623840 13718 consensus_queue.cc:237] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } }
I20260928 14:36:21.624017 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.623763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:40471)
0928 14:36:21.623871 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.623920 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:36:21.626214 13156 catalog_manager.cc:5495] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b174366a0a6848f0ac1bfdc4e4b3b7d1 (127.6.23.195). New cstate: current_term: 1 leader_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b174366a0a6848f0ac1bfdc4e4b3b7d1" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 39693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 } health_report { overall_health: UNKNOWN } } }
I20260928 14:36:21.682529 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.679093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49370 (local address 127.6.23.195:39693)
0928 14:36:21.679353 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:21.679365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:21.679379 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:36:21.679478 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:21.679480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:21.679536 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:21.679637 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:21.679866 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.680360 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.681160 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:21.681365 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:21.681456 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:21.681673 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:21.682127 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:21.682214 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:21.682271 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:36:21.682318 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260928 14:36:21.685659 13289 raft_consensus.cc:1261] T c1c06f38aaff450996fc68cf725fe65d P 0c88a588d8ee4415b59b404a7f1e599e [term 1 FOLLOWER]: Refusing update from remote peer b174366a0a6848f0ac1bfdc4e4b3b7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:21.685909 13426 raft_consensus.cc:1261] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 FOLLOWER]: Refusing update from remote peer b174366a0a6848f0ac1bfdc4e4b3b7d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:21.686264 13718 consensus_queue.cc:1035] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c88a588d8ee4415b59b404a7f1e599e" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:21.686614 13721 consensus_queue.cc:1035] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1c4b0dc62f4bae97b75b3cfe7a0f18" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 40705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:21.688197 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.687907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:35665)
0928 14:36:21.688061 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.688121 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 14:36:21.689587 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.689283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:58537)
0928 14:36:21.689440 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.689492 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 14:36:21.691682 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.691397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:41581)
0928 14:36:21.691553 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.691609 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 14:36:21.693051 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.692797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.194:46459)
0928 14:36:21.692934 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.692983 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 14:36:21.693460 13727 mvcc.cc:205] Tried to move back new op lower bound from 7334322920178724864 to 7334322919933935616. Current Snapshot: MvccSnapshot[applied={T|T < 7334322920178724864 or (T in {7334322920178724864})}]
I20260928 14:36:21.694281 13720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.694052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.195:41305)
0928 14:36:21.694166 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.694220 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:36:21.695170 13725 mvcc.cc:205] Tried to move back new op lower bound from 7334322920178724864 to 7334322919933935616. Current Snapshot: MvccSnapshot[applied={T|T < 7334322920178724864}]
I20260928 14:36:21.696792 13691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.696495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:43803 (local address 127.6.23.193:36431)
0928 14:36:21.696654 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.696713 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:43803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 14:36:21.701936 13156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:21.702104 13156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:21.706002 13156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c0759c6a8cb49569be2f17b9999c360: Sending DeleteTablet for 3 replicas of tablet c1c06f38aaff450996fc68cf725fe65d
I20260928 14:36:21.706745 13406 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c06f38aaff450996fc68cf725fe65d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:21 UTC) from {username='oryx', principal='oryx/127.6.23.252@KRBTEST.COM'} at 127.0.0.1:41982
I20260928 14:36:21.706811 13542 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c06f38aaff450996fc68cf725fe65d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:21 UTC) from {username='oryx', principal='oryx/127.6.23.252@KRBTEST.COM'} at 127.0.0.1:49356
I20260928 14:36:21.706862 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c06f38aaff450996fc68cf725fe65d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:21 UTC) from {username='oryx', principal='oryx/127.6.23.252@KRBTEST.COM'} at 127.0.0.1:39680
I20260928 14:36:21.707160 13735 tablet_replica.cc:330] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: stopping tablet replica
I20260928 14:36:21.707365 13735 raft_consensus.cc:2229] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:36:21.707736 13735 raft_consensus.cc:2258] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:21.707909 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 13116
I20260928 14:36:21.709372 13735 ts_tablet_manager.cc:1905] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:21.711637 13735 ts_tablet_manager.cc:1918] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:21.711732 13735 log.cc:1196] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/c1c06f38aaff450996fc68cf725fe65d
I20260928 14:36:21.712057 13735 ts_tablet_manager.cc:1939] T c1c06f38aaff450996fc68cf725fe65d P b174366a0a6848f0ac1bfdc4e4b3b7d1: Deleting consensus metadata
I20260928 14:36:21.712809 13141 catalog_manager.cc:4841] TS b174366a0a6848f0ac1bfdc4e4b3b7d1 (127.6.23.195:39693): tablet c1c06f38aaff450996fc68cf725fe65d (table test-table [id=ed405857484c4a06886f1bc54b1bb64c]) successfully deleted
I20260928 14:36:21.715360 13734 tablet_replica.cc:330] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: stopping tablet replica
I20260928 14:36:21.715536 13734 raft_consensus.cc:2229] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:21.715705 13734 raft_consensus.cc:2258] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:21.716702 13734 ts_tablet_manager.cc:1905] T c1c06f38aaff450996fc68cf725fe65d P 8f1c4b0dc62f4bae97b75b3cfe7a0f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:21.717653 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 13343
I20260928 14:36:21.718077 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.717746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:37771 (local address 127.0.0.1:39684)
0928 14:36:21.717900 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.717974 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:37771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 14:36:21.718362 13141 catalog_manager.cc:4584] TS 0c88a588d8ee4415b59b404a7f1e599e (127.6.23.193:37771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1c06f38aaff450996fc68cf725fe65d: Network error: Client connection negotiation failed: client connection to 127.6.23.193:37771: connect: Connection refused (error 111)
I20260928 14:36:21.725323 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 13479
I20260928 14:36:21.725680 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:21.725380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:40705 (local address 127.0.0.1:41998)
0928 14:36:21.725555 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:21.725600 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.194:40705: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 14:36:21.730504 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 12835
I20260928 14:36:21.736235 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 12943
2026-09-28T14:36:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7618 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:21 dist-test-slave-06lh krb5kdc[13739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:21 dist-test-slave-06lh krb5kdc[13739](info): set up 2 sockets
Sep 28 14:36:21 dist-test-slave-06lh krb5kdc[13739](info): commencing operation
krb5kdc: starting...
W20260928 14:36:22.780246 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:22 dist-test-slave-06lh krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.254.keytab.
Entry for principal oryx/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.254.keytab.
I20260928 14:36:22.902379 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:33523
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.254.keytab
--principal=oryx/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:33523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:23.018244 13752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:23.018518 13752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:23.018559 13752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:23.022552 13752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:23.022629 13752 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:36:23.022647 13752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:23.022665 13752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:23.022681 13752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:23.022697 13752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:23.028048 13752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:33523
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.254.keytab
--principal=oryx/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:23.029089 13752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:23.030066 13752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:36:23.039136 13752 server_base.cc:1034] running on GCE node
W20260928 14:36:23.039222 13760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:23.039430 13758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:23.039270 13757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:23.040125 13752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:23.041249 13752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:23.042451 13752 hybrid_clock.cc:648] HybridClock initialized: now 1790606183042432 us; error 32 us; skew 500 ppm
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:23.052268 13752 init.cc:377] Logged in from keytab as oryx/127.6.23.254@KRBTEST.COM (short username oryx)
I20260928 14:36:23.053345 13752 webserver.cc:460] Webserver started at http://127.6.23.254:39097/ using document root <none> and password file <none>
I20260928 14:36:23.053970 13752 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:23.054068 13752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:23.054311 13752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:23.056547 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "b1bf86f0785b4cbd9efbad35f48695b2"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "78e1d53a236a7aeade6c88e782f98774"
server_key_iv: "09b79ec4d70ec43d3e61798c6b8e120c"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.057144 13752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "b1bf86f0785b4cbd9efbad35f48695b2"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "78e1d53a236a7aeade6c88e782f98774"
server_key_iv: "09b79ec4d70ec43d3e61798c6b8e120c"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.061072 13752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 14:36:23.063625 13767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:23.064920 13752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:23.065078 13752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "b1bf86f0785b4cbd9efbad35f48695b2"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "78e1d53a236a7aeade6c88e782f98774"
server_key_iv: "09b79ec4d70ec43d3e61798c6b8e120c"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.065197 13752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:23.080140 13752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:23.082998 13752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:23.083209 13752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:23.091351 13752 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:33523
I20260928 14:36:23.091382 13829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:33523 every 8 connection(s)
I20260928 14:36:23.092290 13752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:23.094985 13830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:23.098953 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 13752
I20260928 14:36:23.099067 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:23.099397 6239 external_mini_cluster.cc:1428] Setting key 52cbff10094050c0f446a2cda8d3ad5e
I20260928 14:36:23.100978 13830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: Bootstrap starting.
I20260928 14:36:23.103343 13830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:23.104110 13830 log.cc:824] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:23.106007 13830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: No bootstrap required, opened a new log
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:23.110090 13830 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bf86f0785b4cbd9efbad35f48695b2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33523 } }
I20260928 14:36:23.110374 13830 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:23.110468 13830 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1bf86f0785b4cbd9efbad35f48695b2, State: Initialized, Role: FOLLOWER
I20260928 14:36:23.110944 13830 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bf86f0785b4cbd9efbad35f48695b2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33523 } }
I20260928 14:36:23.111074 13830 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:23.111164 13830 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:23.111297 13830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:23.112308 13830 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bf86f0785b4cbd9efbad35f48695b2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33523 } }
I20260928 14:36:23.112661 13830 leader_election.cc:302] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1bf86f0785b4cbd9efbad35f48695b2; no voters:
I20260928 14:36:23.112982 13830 leader_election.cc:288] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:23.113085 13835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:23.113343 13835 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [term 1 LEADER]: Becoming Leader. State: Replica: b1bf86f0785b4cbd9efbad35f48695b2, State: Running, Role: LEADER
I20260928 14:36:23.113750 13835 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bf86f0785b4cbd9efbad35f48695b2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33523 } }
I20260928 14:36:23.114290 13830 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:23.114888 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.100917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.6.23.254:33523)
0928 14:36:23.101366 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.101381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.101418 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:36:23.102105 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.102122 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.102463 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.102809 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.103724 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.104790 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.105683 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.106216 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.106381 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.109152 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:23.109156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:23.109171 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:23.109198 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:23.111162 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.111688 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.111694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.111700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.111827 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.112164 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.112168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.112170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.112648 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:23.112864 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.113569 (+ 705us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.113888 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":144,"threads_started":1}
I20260928 14:36:23.115454 13836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1bf86f0785b4cbd9efbad35f48695b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bf86f0785b4cbd9efbad35f48695b2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33523 } } }
I20260928 14:36:23.115672 13836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:23.115608 13835 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1bf86f0785b4cbd9efbad35f48695b2. Latest consensus state: current_term: 1 leader_uuid: "b1bf86f0785b4cbd9efbad35f48695b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1bf86f0785b4cbd9efbad35f48695b2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33523 } } }
I20260928 14:36:23.116044 13835 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:23.116353 13844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:23.119740 13844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:23.125926 13844 catalog_manager.cc:1347] Generated new cluster ID: 9a1f566db11c4067887cec0568cb5181
I20260928 14:36:23.125998 13844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:23.142896 13844 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:23.144345 13844 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:23.162958 13844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: Generated new TSK 0
I20260928 14:36:23.163941 13844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab.
Entry for principal oryx/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab.
I20260928 14:36:23.198431 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33523
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab
--principal=oryx/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:23.313009 13856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:23.313277 13856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:23.313324 13856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:23.317806 13856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:23.317873 13856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:23.317914 13856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:36:23.317952 13856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:23.323246 13856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab
--principal=oryx/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:23.324429 13856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:23.325412 13856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:23.335258 13864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:23.335294 13861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:23.335549 13856 server_base.cc:1034] running on GCE node
W20260928 14:36:23.335294 13862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:23.336145 13856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:23.336774 13856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:23.337962 13856 hybrid_clock.cc:648] HybridClock initialized: now 1790606183337927 us; error 49 us; skew 500 ppm
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:23.340977 13856 init.cc:377] Logged in from keytab as oryx/127.6.23.193@KRBTEST.COM (short username oryx)
I20260928 14:36:23.341884 13856 webserver.cc:460] Webserver started at http://127.6.23.193:45793/ using document root <none> and password file <none>
I20260928 14:36:23.342440 13856 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:23.342501 13856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:23.342672 13856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:23.344841 13856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "70a8ed92a63f432eb6ce532d001b55d6"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "4851d70ffb1a4ff6c262ea76548d37e5"
server_key_iv: "084a5e4de26c776c18ed960424df538b"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.345386 13856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "70a8ed92a63f432eb6ce532d001b55d6"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "4851d70ffb1a4ff6c262ea76548d37e5"
server_key_iv: "084a5e4de26c776c18ed960424df538b"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.349225 13856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:23.351768 13871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:23.353048 13856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:23.353180 13856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "70a8ed92a63f432eb6ce532d001b55d6"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "4851d70ffb1a4ff6c262ea76548d37e5"
server_key_iv: "084a5e4de26c776c18ed960424df538b"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.353296 13856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:23.371054 13856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:23.374511 13856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:23.374725 13856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:23.376345 13856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:23.376397 13856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:23.376463 13856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:23.376495 13856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:23.387338 13856 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:38011
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:23.387403 13986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:38011 every 8 connection(s)
I20260928 14:36:23.388401 13856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:23.391608 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.377413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:56315 (local address 127.6.23.254:33523)
0928 14:36:23.377577 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.377582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.378456 (+ 874us) server_negotiation.cc:407] Connection header received
0928 14:36:23.379658 (+ 1202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.379661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.379720 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.379828 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.381466 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.382400 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.384261 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.384662 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.384781 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.388071 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:23.388076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:23.388078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:23.388108 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:23.389749 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.390223 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.390225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.390226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.390301 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.390624 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.390626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.390627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.390810 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:23.390888 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.391368 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.391475 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:36:23.392427 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.377808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33523 (local address 127.6.23.193:56315)
0928 14:36:23.378276 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.378328 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.379416 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.380041 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.380050 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.380524 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:23.381280 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.381302 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.382551 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.382555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.384122 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.384133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.384880 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.384884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.385097 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.385878 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:23.385907 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:23.387879 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:23.389849 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.389853 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.389865 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.390124 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.390377 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.390379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.390381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.390536 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.390900 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:23.390911 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:23.391134 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.391299 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.391580 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":145,"threads_started":1}
I20260928 14:36:23.393657 13987 heartbeater.cc:347] Connected to a master server at 127.6.23.254:33523
I20260928 14:36:23.393795 13987 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:23.394392 13987 heartbeater.cc:511] Master 127.6.23.254:33523 requested a full tablet report, sending...
I20260928 14:36:23.395211 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 13856
I20260928 14:36:23.395399 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:23.395737 6239 external_mini_cluster.cc:1428] Setting key 627bfd25d13065dce848c05c7ea71dcf
I20260928 14:36:23.396270 13783 ts_manager.cc:194] Registered new tserver with Master: 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011)
I20260928 14:36:23.397837 13783 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:56315
I20260928 14:36:23.403738 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.396245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:51039 (local address 127.6.23.254:33523)
0928 14:36:23.396375 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.396382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.396633 (+ 251us) server_negotiation.cc:407] Connection header received
0928 14:36:23.396741 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.396744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.396784 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.396866 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.397367 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.397882 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.398888 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.399299 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.399425 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.400568 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:23.400571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:23.400574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:23.400598 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:23.402086 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.402512 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.402516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.402517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.402597 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.402870 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.402875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.402877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.403058 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:23.403163 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.403446 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.403564 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:36:23.403748 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.396262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33523 (local address 127.6.23.193:51039)
0928 14:36:23.396538 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.396552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.396652 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.396898 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.396901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.397124 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:23.397262 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.397271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.398015 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.398019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.398697 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.398706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.399435 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.399438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.399535 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.400077 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:23.400088 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:23.400423 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:23.402194 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.402199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.402202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.402409 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.402687 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.402692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.402694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.402780 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.403154 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:23.403157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:23.403269 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.403390 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.403567 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for oryx/127.6.23.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab.
Entry for principal oryx/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab.
I20260928 14:36:23.423847 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33523
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab
--principal=oryx/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:23.535984 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:23.536265 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:23.536309 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:23.540336 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:23.540403 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:23.540441 13996 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:36:23.540474 13996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:23.546353 13996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab
--principal=oryx/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:23.547535 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:23.548588 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:23.558537 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:23.558537 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:23.558558 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:23.558784 13996 server_base.cc:1034] running on GCE node
I20260928 14:36:23.559355 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:23.560014 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:23.561209 13996 hybrid_clock.cc:648] HybridClock initialized: now 1790606183561192 us; error 28 us; skew 500 ppm
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:23.564065 13996 init.cc:377] Logged in from keytab as oryx/127.6.23.194@KRBTEST.COM (short username oryx)
I20260928 14:36:23.564952 13996 webserver.cc:460] Webserver started at http://127.6.23.194:42889/ using document root <none> and password file <none>
I20260928 14:36:23.565546 13996 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:23.565639 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:23.565874 13996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:23.568076 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "ae4e881d45b1478ebb6ffdd0eb67e591"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "95734269447b0bceb71abb275a7364a4"
server_key_iv: "61d8b40922eefbfb33612b321832cbb1"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.568702 13996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "ae4e881d45b1478ebb6ffdd0eb67e591"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "95734269447b0bceb71abb275a7364a4"
server_key_iv: "61d8b40922eefbfb33612b321832cbb1"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.572700 13996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:36:23.575213 14011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:23.576527 13996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:23.576673 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "ae4e881d45b1478ebb6ffdd0eb67e591"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "95734269447b0bceb71abb275a7364a4"
server_key_iv: "61d8b40922eefbfb33612b321832cbb1"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.576793 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:23.587929 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:23.591202 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:23.591449 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:23.592849 13996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:23.592921 13996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:23.592999 13996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:23.593042 13996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:23.604317 13996 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:37301
I20260928 14:36:23.604347 14126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:37301 every 8 connection(s)
I20260928 14:36:23.605259 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:23.607417 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.593787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:50455 (local address 127.6.23.254:33523)
0928 14:36:23.593913 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.593917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.595064 (+ 1147us) server_negotiation.cc:407] Connection header received
0928 14:36:23.596263 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.596267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.596330 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.596440 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.597974 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.598884 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.600281 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.600578 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.600690 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.603505 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:23.603509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:23.603511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:23.603537 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:23.605219 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.605819 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.605824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.605832 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.605950 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.606348 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.606351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.606353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.606537 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:23.606619 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.607093 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.607208 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:36:23.608148 14020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.594373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33523 (local address 127.6.23.194:50455)
0928 14:36:23.594898 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.594941 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.595990 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.596622 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.596634 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.597100 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:23.597821 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.597844 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.598999 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.599003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.600151 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.600161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.600742 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.600746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.600922 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.601511 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:23.601527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:23.603333 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:23.605337 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.605342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.605356 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.605688 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.606064 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.606067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.606070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.606218 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.606629 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:23.606634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:23.606844 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.607000 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.607291 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":137,"threads_started":1}
I20260928 14:36:23.609495 14127 heartbeater.cc:347] Connected to a master server at 127.6.23.254:33523
I20260928 14:36:23.609678 14127 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:23.610188 14127 heartbeater.cc:511] Master 127.6.23.254:33523 requested a full tablet report, sending...
I20260928 14:36:23.610654 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 13996
I20260928 14:36:23.610780 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:23.611109 6239 external_mini_cluster.cc:1428] Setting key bf5968436e5121e49d30910d70594e8e
I20260928 14:36:23.611186 13784 ts_manager.cc:194] Registered new tserver with Master: ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301)
I20260928 14:36:23.611974 13784 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:50455
I20260928 14:36:23.620573 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.612085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:51169 (local address 127.6.23.254:33523)
0928 14:36:23.612213 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.612218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.612229 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:36:23.612263 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.612267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.612311 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.612372 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.613114 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.613922 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.615287 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.615678 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.615969 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.617168 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:23.617173 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:23.617178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:23.617215 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:23.618716 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.619226 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.619259 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.619286 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.619386 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.619735 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.619738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.619739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.619949 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:23.620043 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.620353 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.620450 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:36:23.620725 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.611747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33523 (local address 127.6.23.194:51169)
0928 14:36:23.612011 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.612027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.612114 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.612554 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.612557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.612766 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:23.612886 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.612894 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.614070 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.614072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.615132 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.615146 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.615871 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.615879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.616021 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.616507 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:23.616527 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:23.616980 (+ 453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:23.618835 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.618840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.618843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.619105 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.619494 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.619497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.619500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.619616 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.620055 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:23.620060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:23.620169 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.620298 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.620470 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for oryx/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab.
Entry for principal oryx/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab.
I20260928 14:36:23.638324 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33523
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab
--principal=oryx/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:23.752239 14136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:23.752558 14136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:23.752640 14136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:23.756732 14136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:23.756822 14136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:23.756893 14136 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:36:23.756966 14136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:23.762387 14136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab
--principal=oryx/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.23.254:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:23.763583 14136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:23.764633 14136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:23.774955 14142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:23.774972 14141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:23.775031 14136 server_base.cc:1034] running on GCE node
W20260928 14:36:23.774955 14144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:23.775691 14136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:23.776365 14136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:23.777566 14136 hybrid_clock.cc:648] HybridClock initialized: now 1790606183777551 us; error 30 us; skew 500 ppm
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:23.780356 14136 init.cc:377] Logged in from keytab as oryx/127.6.23.195@KRBTEST.COM (short username oryx)
I20260928 14:36:23.781423 14136 webserver.cc:460] Webserver started at http://127.6.23.195:35525/ using document root <none> and password file <none>
I20260928 14:36:23.781996 14136 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:23.782083 14136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:23.782315 14136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:23.784469 14136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "1cb8820f933d453fac85ab89fe98a732"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "85a05e68ebe4f5560066efdb2c81899e"
server_key_iv: "4c690ea46c0c59e563a333f10ba53d00"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.785037 14136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "1cb8820f933d453fac85ab89fe98a732"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "85a05e68ebe4f5560066efdb2c81899e"
server_key_iv: "4c690ea46c0c59e563a333f10ba53d00"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.788810 14136 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:36:23.791312 14151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:23.792549 14136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:36:23.792690 14136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "1cb8820f933d453fac85ab89fe98a732"
format_stamp: "Formatted at 2026-09-28 14:36:23 on dist-test-slave-06lh"
server_key: "85a05e68ebe4f5560066efdb2c81899e"
server_key_iv: "4c690ea46c0c59e563a333f10ba53d00"
server_key_version: "encryptionkey@0"
I20260928 14:36:23.792807 14136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:23.814873 14136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:23.818212 14136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:23.818435 14136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:23.820056 14136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:23.820134 14136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:23.820210 14136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:23.820254 14136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:23.831388 14136 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:44359
I20260928 14:36:23.831507 14266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:44359 every 8 connection(s)
I20260928 14:36:23.832607 14136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:23.834179 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.820842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:59389 (local address 127.6.23.254:33523)
0928 14:36:23.821016 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.821022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.821913 (+ 891us) server_negotiation.cc:407] Connection header received
0928 14:36:23.822982 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.822987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.823045 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.823108 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.824824 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.825477 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.826829 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.827121 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.827785 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.830296 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:23.830301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:23.830303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:23.830331 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:23.832104 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.832608 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.832612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.832614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.832711 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.833120 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.833123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.833125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.833348 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:23.833449 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.833952 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.834058 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:36:23.834942 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 14136
I20260928 14:36:23.835034 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.821282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33523 (local address 127.6.23.195:59389)
0928 14:36:23.821742 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.821792 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.822718 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.823584 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.823592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.824025 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:23.824666 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.824686 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.825607 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.825610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.826701 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.826711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.827361 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.827364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.827510 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.828147 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:23.828171 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:23.830112 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:23.832220 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.832225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.832236 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.832496 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.832822 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.832825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.832826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.832998 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.833447 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:23.833453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:23.833736 (+ 283us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.833897 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.834158 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":142,"threads_started":1}
I20260928 14:36:23.835070 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:23.835424 6239 external_mini_cluster.cc:1428] Setting key af8a7442c1cedf7c2a4cc5f106aba3b4
I20260928 14:36:23.836527 14267 heartbeater.cc:347] Connected to a master server at 127.6.23.254:33523
I20260928 14:36:23.836666 14267 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:23.837262 14267 heartbeater.cc:511] Master 127.6.23.254:33523 requested a full tablet report, sending...
I20260928 14:36:23.838274 13783 ts_manager.cc:194] Registered new tserver with Master: 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:23.839046 13783 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:59389
I20260928 14:36:23.845196 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.838532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33523 (local address 127.6.23.195:58393)
0928 14:36:23.838839 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.838851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.838926 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.839154 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.839156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.839426 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:23.839524 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.839529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.840293 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.840296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.841111 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.841119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.841518 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.841520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.841589 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.842118 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:23.842129 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:23.842435 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:23.844009 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.844011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.844013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.844193 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.844421 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:23.844423 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:23.844424 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:23.844495 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:23.844805 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:23.844807 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:23.844882 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.844963 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.845065 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":87,"threads_started":1}
I20260928 14:36:23.845216 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.838633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:58393 (local address 127.6.23.254:33523)
0928 14:36:23.838758 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.838763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.838913 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:36:23.839003 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.839006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.839050 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.839133 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.839637 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.840125 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.841212 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.841413 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.841494 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.842544 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:23.842546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:23.842548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:23.842564 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:23.843913 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.844275 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.844277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.844278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.844346 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.844569 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.844571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.844572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.844723 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:23.844795 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.845020 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.845108 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:36:23.849187 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:36:23.872567 13833 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:23.866139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.6.23.254:33523)
0928 14:36:23.866271 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.866274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.866285 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:36:23.866326 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.866328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.866367 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.866470 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.867172 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.867779 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.868760 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.868994 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.869104 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.872395 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51872: BlockingRecv error: recv got EOF from 127.0.0.1:51872 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 14:36:23 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:23.882198 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.873985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51882 (local address 127.6.23.254:33523)
0928 14:36:23.874121 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.874126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.874253 (+ 127us) server_negotiation.cc:407] Connection header received
0928 14:36:23.874405 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.874410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.874468 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.874552 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:23.875143 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.875721 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.876534 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.876756 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.876866 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.879076 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:23.879079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:23.879081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:23.879105 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:23.880673 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.881093 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.881096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.881097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.881168 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:23.881412 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:23.881415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:23.881416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:23.881594 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:23.881695 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.881886 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.881990 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:36:23.885082 13783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:36:23.888048 13783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:23.902771 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.897093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.6.23.194:37301)
0928 14:36:23.897919 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.897929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.897947 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:36:23.898008 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.898013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.898164 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.898316 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:23.898524 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.899367 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.901076 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.901908 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.902027 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.902190 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.902521 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.902618 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":97,"threads_started":1}
I20260928 14:36:23.902786 14292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.896483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37301 (local address 127.0.0.1:55520)
0928 14:36:23.897381 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.897395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.897541 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.898297 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.898300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.898317 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:23.898419 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.898425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.899481 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.899488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.900972 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.900980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.902038 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.902042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.902292 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.902352 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.902474 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.902644 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":198,"thread_start_us":105,"threads_started":1}
I20260928 14:36:23.902915 14294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.897122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.6.23.195:44359)
0928 14:36:23.897841 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.897854 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.897878 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:36:23.897971 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.897977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.898165 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.898316 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:23.898743 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.899874 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.901148 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.901908 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.902027 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.902190 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.902676 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.902772 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":111,"threads_started":1}
I20260928 14:36:23.903224 14291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.896769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:44359 (local address 127.0.0.1:34954)
0928 14:36:23.897505 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.897517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.897586 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.898495 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.898498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.898519 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:23.898614 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.898620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.900016 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.900020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.901017 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.901027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.902038 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.902042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.902287 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.902352 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.902476 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.903077 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":551,"thread_start_us":82,"threads_started":1}
I20260928 14:36:23.903628 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.896314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38011 (local address 127.0.0.1:48688)
0928 14:36:23.897138 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:23.897190 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:23.897643 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:23.898109 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:23.898115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:23.898136 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:23.898288 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.898296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.899728 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.899731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.900969 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:23.900980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.903150 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.903155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:23.903293 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.903323 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:36:23.903406 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:36:23.903486 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":31,"thread_start_us":89,"threads_started":1}
I20260928 14:36:23.903700 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:23.896478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.6.23.193:38011)
0928 14:36:23.897076 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:23.897083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:23.897569 (+ 486us) server_negotiation.cc:407] Connection header received
0928 14:36:23.897800 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:23.897805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:23.897972 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:23.898121 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:23.898413 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.899617 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.902257 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:23.903035 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:23.903133 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:23.903286 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:23.903456 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 14:36:23.903564 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":80,"threads_started":1}
I20260928 14:36:23.906541 13921 tablet_service.cc:1467] Processing CreateTablet for tablet ee23150e873c472ab913e499964f43a6 (DEFAULT_TABLE table=test-table [id=10869bb5e12a4d2bbce9a34095e74611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:23.906554 14201 tablet_service.cc:1467] Processing CreateTablet for tablet ee23150e873c472ab913e499964f43a6 (DEFAULT_TABLE table=test-table [id=10869bb5e12a4d2bbce9a34095e74611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:23.906623 14061 tablet_service.cc:1467] Processing CreateTablet for tablet ee23150e873c472ab913e499964f43a6 (DEFAULT_TABLE table=test-table [id=10869bb5e12a4d2bbce9a34095e74611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:23.907635 14061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee23150e873c472ab913e499964f43a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:23.907653 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee23150e873c472ab913e499964f43a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:23.907653 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee23150e873c472ab913e499964f43a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:23.914871 14295 tablet_bootstrap.cc:492] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Bootstrap starting.
I20260928 14:36:23.915402 14297 tablet_bootstrap.cc:492] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Bootstrap starting.
I20260928 14:36:23.915898 14296 tablet_bootstrap.cc:492] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Bootstrap starting.
I20260928 14:36:23.917291 14295 tablet_bootstrap.cc:654] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:23.917621 14297 tablet_bootstrap.cc:654] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:23.917873 14296 tablet_bootstrap.cc:654] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:23.918128 14295 log.cc:824] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:23.918450 14297 log.cc:824] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:23.918589 14296 log.cc:824] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:23.920226 14295 tablet_bootstrap.cc:492] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: No bootstrap required, opened a new log
I20260928 14:36:23.920365 14297 tablet_bootstrap.cc:492] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: No bootstrap required, opened a new log
I20260928 14:36:23.920491 14295 ts_tablet_manager.cc:1397] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:36:23.920506 14297 ts_tablet_manager.cc:1397] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:36:23.920811 14296 tablet_bootstrap.cc:492] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: No bootstrap required, opened a new log
I20260928 14:36:23.920941 14296 ts_tablet_manager.cc:1397] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:36:23.924105 14296 raft_consensus.cc:348] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:23.924366 14296 raft_consensus.cc:374] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:23.924449 14296 raft_consensus.cc:729] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a8ed92a63f432eb6ce532d001b55d6, State: Initialized, Role: FOLLOWER
I20260928 14:36:23.924535 14295 raft_consensus.cc:348] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:23.924535 14297 raft_consensus.cc:348] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:23.924787 14297 raft_consensus.cc:374] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:23.924831 14295 raft_consensus.cc:374] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:23.924866 14297 raft_consensus.cc:729] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb8820f933d453fac85ab89fe98a732, State: Initialized, Role: FOLLOWER
I20260928 14:36:23.924901 14295 raft_consensus.cc:729] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e881d45b1478ebb6ffdd0eb67e591, State: Initialized, Role: FOLLOWER
I20260928 14:36:23.924990 14296 consensus_queue.cc:260] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:23.925346 14297 consensus_queue.cc:260] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:23.925346 14295 consensus_queue.cc:260] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:23.925726 13987 heartbeater.cc:503] Master 127.6.23.254:33523 was elected leader, sending a full tablet report...
I20260928 14:36:23.926000 14127 heartbeater.cc:503] Master 127.6.23.254:33523 was elected leader, sending a full tablet report...
I20260928 14:36:23.926220 14267 heartbeater.cc:503] Master 127.6.23.254:33523 was elected leader, sending a full tablet report...
I20260928 14:36:23.926412 14297 ts_tablet_manager.cc:1428] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:36:23.927616 14295 ts_tablet_manager.cc:1428] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260928 14:36:23.927768 14296 ts_tablet_manager.cc:1428] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260928 14:36:24.083631 14268 tablet.cc:2369] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:36:24.106648 14128 tablet.cc:2369] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:24.131781 14301 raft_consensus.cc:482] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:24.132102 14301 raft_consensus.cc:504] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.133530 14301 leader_election.cc:288] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301), 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:24.137729 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.133830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:58861 (local address 127.6.23.194:37301)
0928 14:36:24.134002 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.134007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.134240 (+ 233us) server_negotiation.cc:407] Connection header received
0928 14:36:24.134444 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.134447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.134505 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.134564 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:24.134918 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.135523 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.136683 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.137130 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.137228 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.137321 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.137544 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.137618 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:36:24.137754 14304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.133719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37301 (local address 127.6.23.193:58861)
0928 14:36:24.134113 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.134129 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.134297 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.134603 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.134606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.134622 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:24.134805 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.134810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.135629 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.135633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.136567 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.136576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.137248 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.137252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.137370 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.137392 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.137500 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.137581 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":73,"spinlock_wait_cycles":29184,"thread_start_us":135,"threads_started":1}
I20260928 14:36:24.137912 14294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.133983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:47493 (local address 127.6.23.195:44359)
0928 14:36:24.134113 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.134118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.134260 (+ 142us) server_negotiation.cc:407] Connection header received
0928 14:36:24.134470 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.134473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.134516 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.134607 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:24.134959 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.135532 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.136818 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.137249 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.137365 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.137416 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.137692 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.137769 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:36:24.137939 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.133826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:44359 (local address 127.6.23.193:47493)
0928 14:36:24.134139 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.134150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.134309 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.134634 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.134636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.134654 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:24.134815 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.134820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.135656 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.135659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.136700 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.136709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.137457 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.137464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.137561 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.137580 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.137665 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.137739 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":48,"spinlock_wait_cycles":2304,"thread_start_us":112,"threads_started":1}
I20260928 14:36:24.138587 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee23150e873c472ab913e499964f43a6" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" is_pre_election: true
I20260928 14:36:24.138723 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee23150e873c472ab913e499964f43a6" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb8820f933d453fac85ab89fe98a732" is_pre_election: true
I20260928 14:36:24.138975 14081 raft_consensus.cc:2454] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 0.
I20260928 14:36:24.139047 14219 raft_consensus.cc:2454] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 0.
I20260928 14:36:24.139494 13874 leader_election.cc:302] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70a8ed92a63f432eb6ce532d001b55d6, ae4e881d45b1478ebb6ffdd0eb67e591; no voters:
W20260928 14:36:24.139822 13988 tablet.cc:2369] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:24.139798 14301 raft_consensus.cc:2781] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:24.139894 14301 raft_consensus.cc:482] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:24.139946 14301 raft_consensus.cc:3037] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.141124 14301 raft_consensus.cc:504] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.141455 14301 leader_election.cc:288] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 election: Requested vote from peers ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301), 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:24.141860 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee23150e873c472ab913e499964f43a6" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb8820f933d453fac85ab89fe98a732"
I20260928 14:36:24.142011 14219 raft_consensus.cc:3037] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.143033 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee23150e873c472ab913e499964f43a6" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591"
I20260928 14:36:24.143177 14081 raft_consensus.cc:3037] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.143613 14219 raft_consensus.cc:2454] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 1.
I20260928 14:36:24.143960 13875 leader_election.cc:302] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb8820f933d453fac85ab89fe98a732, 70a8ed92a63f432eb6ce532d001b55d6; no voters:
I20260928 14:36:24.144165 14301 raft_consensus.cc:2781] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:24.144407 14301 raft_consensus.cc:686] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 LEADER]: Becoming Leader. State: Replica: 70a8ed92a63f432eb6ce532d001b55d6, State: Running, Role: LEADER
I20260928 14:36:24.144574 14081 raft_consensus.cc:2454] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 1.
I20260928 14:36:24.144801 14301 consensus_queue.cc:237] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.148252 13782 catalog_manager.cc:5495] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "70a8ed92a63f432eb6ce532d001b55d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } health_report { overall_health: UNKNOWN } } }
I20260928 14:36:24.197889 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.194263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48690 (local address 127.6.23.193:38011)
0928 14:36:24.194412 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.194416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.194433 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:36:24.194528 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.194533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.194599 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.194716 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:24.194962 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.195501 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.196393 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.196626 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.196739 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.197022 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.197474 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.197621 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.197684 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.197741 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:36:24.203743 14081 raft_consensus.cc:1261] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Refusing update from remote peer 70a8ed92a63f432eb6ce532d001b55d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.203778 14219 raft_consensus.cc:1261] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Refusing update from remote peer 70a8ed92a63f432eb6ce532d001b55d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.204473 14306 consensus_queue.cc:1035] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.204677 14301 consensus_queue.cc:1035] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.214388 14311 mvcc.cc:205] Tried to move back new op lower bound from 7334322930491305984 to 7334322930260115456. Current Snapshot: MvccSnapshot[applied={T|T < 7334322930491305984}]
I20260928 14:36:24.224597 13782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:24.224817 13782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:24.227468 13782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: Sending DeleteTablet for 3 replicas of tablet ee23150e873c472ab913e499964f43a6
I20260928 14:36:24.228307 13921 tablet_service.cc:1514] Processing DeleteTablet for tablet ee23150e873c472ab913e499964f43a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:48688
I20260928 14:36:24.228305 14201 tablet_service.cc:1514] Processing DeleteTablet for tablet ee23150e873c472ab913e499964f43a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34954
I20260928 14:36:24.228864 14319 tablet_replica.cc:330] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: stopping tablet replica
I20260928 14:36:24.228858 14061 tablet_service.cc:1514] Processing DeleteTablet for tablet ee23150e873c472ab913e499964f43a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:55520
I20260928 14:36:24.229147 13782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:36:24.229285 14321 tablet_replica.cc:330] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: stopping tablet replica
I20260928 14:36:24.229547 14319 raft_consensus.cc:2229] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:24.229576 14321 raft_consensus.cc:2229] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 14:36:24.229590 13782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:24.229951 14319 raft_consensus.cc:2258] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.229956 14321 raft_consensus.cc:2258] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.231560 14321 ts_tablet_manager.cc:1905] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.232384 14319 ts_tablet_manager.cc:1905] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.234411 14321 ts_tablet_manager.cc:1918] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:24.234531 14321 log.cc:1196] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/ee23150e873c472ab913e499964f43a6
I20260928 14:36:24.234850 14321 ts_tablet_manager.cc:1939] T ee23150e873c472ab913e499964f43a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting consensus metadata
I20260928 14:36:24.235132 13921 tablet_service.cc:1467] Processing CreateTablet for tablet 5c93927ffaa84f01944710b68c4911a6 (DEFAULT_TABLE table=test-table [id=2613f7f04b6e4359993f068500f0a327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.235386 14061 tablet_service.cc:1467] Processing CreateTablet for tablet 5c93927ffaa84f01944710b68c4911a6 (DEFAULT_TABLE table=test-table [id=2613f7f04b6e4359993f068500f0a327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.235442 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c93927ffaa84f01944710b68c4911a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.235409 14201 tablet_service.cc:1467] Processing CreateTablet for tablet 5c93927ffaa84f01944710b68c4911a6 (DEFAULT_TABLE table=test-table [id=2613f7f04b6e4359993f068500f0a327]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.235615 14061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c93927ffaa84f01944710b68c4911a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.235641 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c93927ffaa84f01944710b68c4911a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.236153 13770 catalog_manager.cc:4841] TS ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301): tablet ee23150e873c472ab913e499964f43a6 (table test-table [id=10869bb5e12a4d2bbce9a34095e74611]) successfully deleted
I20260928 14:36:24.236651 14320 tablet_replica.cc:330] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: stopping tablet replica
I20260928 14:36:24.236999 14320 raft_consensus.cc:2229] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:36:24.237407 14320 raft_consensus.cc:2258] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.237856 14295 tablet_bootstrap.cc:492] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Bootstrap starting.
I20260928 14:36:24.237887 14296 tablet_bootstrap.cc:492] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: Bootstrap starting.
I20260928 14:36:24.237977 14297 tablet_bootstrap.cc:492] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: Bootstrap starting.
I20260928 14:36:24.238821 14295 tablet_bootstrap.cc:654] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.238888 14297 tablet_bootstrap.cc:654] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.238888 14296 tablet_bootstrap.cc:654] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.240167 14296 tablet_bootstrap.cc:492] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: No bootstrap required, opened a new log
I20260928 14:36:24.240223 14319 ts_tablet_manager.cc:1918] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:24.240255 14296 ts_tablet_manager.cc:1397] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:36:24.240309 14319 log.cc:1196] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/ee23150e873c472ab913e499964f43a6
I20260928 14:36:24.240429 14297 tablet_bootstrap.cc:492] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: No bootstrap required, opened a new log
I20260928 14:36:24.240513 14297 ts_tablet_manager.cc:1397] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 14:36:24.240633 14319 ts_tablet_manager.cc:1939] T ee23150e873c472ab913e499964f43a6 P 1cb8820f933d453fac85ab89fe98a732: Deleting consensus metadata
I20260928 14:36:24.240751 14296 raft_consensus.cc:348] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.240869 14296 raft_consensus.cc:374] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.240906 14296 raft_consensus.cc:729] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a8ed92a63f432eb6ce532d001b55d6, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.240958 14297 raft_consensus.cc:348] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.241056 14297 raft_consensus.cc:374] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.241035 14296 consensus_queue.cc:260] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.241086 14297 raft_consensus.cc:729] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb8820f933d453fac85ab89fe98a732, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.241197 14297 consensus_queue.cc:260] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.241421 14297 ts_tablet_manager.cc:1428] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:36:24.242404 13771 catalog_manager.cc:4841] TS 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359): tablet ee23150e873c472ab913e499964f43a6 (table test-table [id=10869bb5e12a4d2bbce9a34095e74611]) successfully deleted
I20260928 14:36:24.242650 14296 ts_tablet_manager.cc:1428] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 14:36:24.243021 14320 ts_tablet_manager.cc:1905] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.245436 14320 ts_tablet_manager.cc:1918] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:24.245554 14320 log.cc:1196] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/ee23150e873c472ab913e499964f43a6
I20260928 14:36:24.245852 14320 ts_tablet_manager.cc:1939] T ee23150e873c472ab913e499964f43a6 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting consensus metadata
I20260928 14:36:24.247073 13768 catalog_manager.cc:4841] TS 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011): tablet ee23150e873c472ab913e499964f43a6 (table test-table [id=10869bb5e12a4d2bbce9a34095e74611]) successfully deleted
I20260928 14:36:24.247844 14295 tablet_bootstrap.cc:492] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: No bootstrap required, opened a new log
I20260928 14:36:24.247952 14295 ts_tablet_manager.cc:1397] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260928 14:36:24.248678 14295 raft_consensus.cc:348] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.248805 14295 raft_consensus.cc:374] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.248836 14295 raft_consensus.cc:729] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e881d45b1478ebb6ffdd0eb67e591, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.248981 14295 consensus_queue.cc:260] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.249260 14295 ts_tablet_manager.cc:1428] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:36:24.314740 14314 raft_consensus.cc:482] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:24.314877 14314 raft_consensus.cc:504] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.315517 14314 leader_election.cc:288] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301), 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:24.316025 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c93927ffaa84f01944710b68c4911a6" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" is_pre_election: true
I20260928 14:36:24.316038 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c93927ffaa84f01944710b68c4911a6" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb8820f933d453fac85ab89fe98a732" is_pre_election: true
I20260928 14:36:24.316200 14081 raft_consensus.cc:2454] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 0.
I20260928 14:36:24.316246 14219 raft_consensus.cc:2454] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 0.
I20260928 14:36:24.316506 13874 leader_election.cc:302] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70a8ed92a63f432eb6ce532d001b55d6, ae4e881d45b1478ebb6ffdd0eb67e591; no voters:
I20260928 14:36:24.316759 14314 raft_consensus.cc:2781] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:24.316828 14314 raft_consensus.cc:482] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:24.316881 14314 raft_consensus.cc:3037] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.317975 14314 raft_consensus.cc:504] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.318375 14314 leader_election.cc:288] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 election: Requested vote from peers ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301), 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:24.318755 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c93927ffaa84f01944710b68c4911a6" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591"
I20260928 14:36:24.318758 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c93927ffaa84f01944710b68c4911a6" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb8820f933d453fac85ab89fe98a732"
I20260928 14:36:24.318902 14081 raft_consensus.cc:3037] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.318897 14219 raft_consensus.cc:3037] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.319922 14081 raft_consensus.cc:2454] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 1.
I20260928 14:36:24.320097 14219 raft_consensus.cc:2454] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 1.
I20260928 14:36:24.320235 13874 leader_election.cc:302] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70a8ed92a63f432eb6ce532d001b55d6, ae4e881d45b1478ebb6ffdd0eb67e591; no voters:
I20260928 14:36:24.320412 14314 raft_consensus.cc:2781] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:24.320516 14314 raft_consensus.cc:686] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 LEADER]: Becoming Leader. State: Replica: 70a8ed92a63f432eb6ce532d001b55d6, State: Running, Role: LEADER
I20260928 14:36:24.320653 14314 consensus_queue.cc:237] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.322394 13783 catalog_manager.cc:5495] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "70a8ed92a63f432eb6ce532d001b55d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:36:24 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606183, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.254@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:24.374538 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.365038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33523 (local address 127.0.0.1:51896)
0928 14:36:24.365346 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.365364 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.365476 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.365769 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.365772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.366235 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:24.366338 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.366345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.367431 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.367434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.367979 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.367986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.368580 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.368586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.368667 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.369327 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:24.369339 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:24.370889 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:24.373004 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:24.373014 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:24.373020 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:24.373250 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:24.373552 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:24.373555 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:24.373556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:24.373635 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:24.374050 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:24.374056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:24.374166 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.374261 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.374379 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":109,"threads_started":1}
I20260928 14:36:24.374637 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.365151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.6.23.254:33523)
0928 14:36:24.365313 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.365318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.365452 (+ 134us) server_negotiation.cc:407] Connection header received
0928 14:36:24.365576 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.365579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.365631 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.365728 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:24.366469 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.367327 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.368089 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.368418 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.368518 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.371034 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:24.371038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:24.371041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:24.371066 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:24.372888 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:24.373346 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:24.373350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:24.373352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:24.373439 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:24.373721 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:24.373724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:24.373726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:24.373946 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:24.374042 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.374330 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.374448 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:36:24.378918 13783 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:51896:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:36:24.379864 13783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:24.385054 13921 tablet_service.cc:1467] Processing CreateTablet for tablet 351b81d34813462998a39d6f2bc0721d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09d9ca096a634c258f046ef0df577558]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:36:24.385341 14201 tablet_service.cc:1467] Processing CreateTablet for tablet 351b81d34813462998a39d6f2bc0721d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09d9ca096a634c258f046ef0df577558]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:36:24.385462 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351b81d34813462998a39d6f2bc0721d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.385716 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351b81d34813462998a39d6f2bc0721d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.388439 14297 tablet_bootstrap.cc:492] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: Bootstrap starting.
I20260928 14:36:24.388569 14061 tablet_service.cc:1467] Processing CreateTablet for tablet 351b81d34813462998a39d6f2bc0721d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=09d9ca096a634c258f046ef0df577558]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:36:24.388913 14061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 351b81d34813462998a39d6f2bc0721d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.389375 14297 tablet_bootstrap.cc:654] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.390887 14297 tablet_bootstrap.cc:492] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: No bootstrap required, opened a new log
I20260928 14:36:24.390967 14297 ts_tablet_manager.cc:1397] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:36:24.391444 14297 raft_consensus.cc:348] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.391575 14297 raft_consensus.cc:374] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.391616 14297 raft_consensus.cc:729] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb8820f933d453fac85ab89fe98a732, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.391757 14297 consensus_queue.cc:260] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.392035 14297 ts_tablet_manager.cc:1428] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:24.392261 14303 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.392395 14303 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:36:24.392414 14295 tablet_bootstrap.cc:492] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: Bootstrap starting.
I20260928 14:36:24.393342 14295 tablet_bootstrap.cc:654] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.394322 14296 tablet_bootstrap.cc:492] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: Bootstrap starting.
I20260928 14:36:24.395213 14296 tablet_bootstrap.cc:654] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.395332 14295 tablet_bootstrap.cc:492] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: No bootstrap required, opened a new log
I20260928 14:36:24.395426 14295 ts_tablet_manager.cc:1397] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:24.395898 14295 raft_consensus.cc:348] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.396034 14295 raft_consensus.cc:374] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.396076 14295 raft_consensus.cc:729] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e881d45b1478ebb6ffdd0eb67e591, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.396203 14295 consensus_queue.cc:260] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.396432 14296 tablet_bootstrap.cc:492] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: No bootstrap required, opened a new log
I20260928 14:36:24.396515 14296 ts_tablet_manager.cc:1397] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:36:24.396499 14295 ts_tablet_manager.cc:1428] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:24.396498 14302 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.396644 14302 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:36:24.396848 14296 raft_consensus.cc:348] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.396946 14296 raft_consensus.cc:374] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.396977 14296 raft_consensus.cc:729] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a8ed92a63f432eb6ce532d001b55d6, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.397080 14296 consensus_queue.cc:260] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.397286 14296 ts_tablet_manager.cc:1428] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:36:24.397464 14314 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.397604 14314 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:36:24.504963 14302 raft_consensus.cc:482] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:24.505131 14302 raft_consensus.cc:504] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.506259 14302 leader_election.cc:288] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011), 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:24.511977 14294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.506757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:32899 (local address 127.6.23.195:44359)
0928 14:36:24.506904 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.506907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.507096 (+ 189us) server_negotiation.cc:407] Connection header received
0928 14:36:24.507275 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.507278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.507343 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.507454 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:24.507928 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.508730 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.510597 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.511257 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.511392 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.511451 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.511757 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.511835 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:36:24.511977 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.506586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:40823 (local address 127.6.23.193:38011)
0928 14:36:24.506731 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.506736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.507097 (+ 361us) server_negotiation.cc:407] Connection header received
0928 14:36:24.507275 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.507278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.507343 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.507437 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:24.507928 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.508716 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.510597 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.511257 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.511393 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.511451 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.511757 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.511835 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:36:24.512040 14336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.506473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38011 (local address 127.6.23.194:40823)
0928 14:36:24.506961 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.506977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.507133 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.507475 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.507478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.507517 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:24.507802 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.507809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.508832 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.508836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.510456 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.510468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.511393 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.511397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.511567 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.511643 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.511726 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.511846 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":318,"spinlock_wait_cycles":44672,"thread_start_us":130,"threads_started":1}
I20260928 14:36:24.512049 14335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.506588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:44359 (local address 127.6.23.194:32899)
0928 14:36:24.506998 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.507014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.507133 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.507475 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.507478 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.507517 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:24.507802 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.507809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.508832 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.508835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.510456 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.510468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.511393 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.511397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.511563 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.511643 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.511726 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.511846 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":259,"spinlock_wait_cycles":29952,"thread_start_us":120,"threads_started":1}
I20260928 14:36:24.512703 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351b81d34813462998a39d6f2bc0721d" candidate_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb8820f933d453fac85ab89fe98a732" is_pre_election: true
I20260928 14:36:24.512712 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351b81d34813462998a39d6f2bc0721d" candidate_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a8ed92a63f432eb6ce532d001b55d6" is_pre_election: true
I20260928 14:36:24.512869 14219 raft_consensus.cc:2454] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4e881d45b1478ebb6ffdd0eb67e591 in term 0.
I20260928 14:36:24.512928 13941 raft_consensus.cc:2454] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae4e881d45b1478ebb6ffdd0eb67e591 in term 0.
I20260928 14:36:24.513295 14015 leader_election.cc:302] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cb8820f933d453fac85ab89fe98a732, 70a8ed92a63f432eb6ce532d001b55d6, ae4e881d45b1478ebb6ffdd0eb67e591; no voters:
I20260928 14:36:24.513576 14302 raft_consensus.cc:2781] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:24.513633 14302 raft_consensus.cc:482] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:24.513669 14302 raft_consensus.cc:3037] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.514712 14302 raft_consensus.cc:504] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.515175 14302 leader_election.cc:288] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [CANDIDATE]: Term 1 election: Requested vote from peers 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011), 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:24.515565 14219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351b81d34813462998a39d6f2bc0721d" candidate_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb8820f933d453fac85ab89fe98a732"
I20260928 14:36:24.515587 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "351b81d34813462998a39d6f2bc0721d" candidate_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a8ed92a63f432eb6ce532d001b55d6"
I20260928 14:36:24.515669 14219 raft_consensus.cc:3037] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.515738 13941 raft_consensus.cc:3037] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.516685 14219 raft_consensus.cc:2454] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4e881d45b1478ebb6ffdd0eb67e591 in term 1.
I20260928 14:36:24.516760 13941 raft_consensus.cc:2454] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae4e881d45b1478ebb6ffdd0eb67e591 in term 1.
I20260928 14:36:24.516970 14015 leader_election.cc:302] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb8820f933d453fac85ab89fe98a732, ae4e881d45b1478ebb6ffdd0eb67e591; no voters:
I20260928 14:36:24.517179 14302 raft_consensus.cc:2781] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:24.517364 14302 raft_consensus.cc:686] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 LEADER]: Becoming Leader. State: Replica: ae4e881d45b1478ebb6ffdd0eb67e591, State: Running, Role: LEADER
I20260928 14:36:24.517657 14302 consensus_queue.cc:237] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.518985 14337 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: TxnStatusTablet state changed. Reason: New leader ae4e881d45b1478ebb6ffdd0eb67e591. Latest consensus state: current_term: 1 leader_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.519107 14337 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:36:24.519376 14339 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:36:24.519541 14339 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:36:24.520242 13782 catalog_manager.cc:5495] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194). New cstate: current_term: 1 leader_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } health_report { overall_health: UNKNOWN } } }
I20260928 14:36:24.553140 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.549275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37301 (local address 127.0.0.1:55522)
0928 14:36:24.549434 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.549452 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.549564 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.549930 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.549933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.549943 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:36:24.550061 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.550066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.550800 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.550805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.551500 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.551510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.551958 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.551962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.552063 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.552108 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.552819 (+ 711us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.552837 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.552921 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.553013 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:36:24.553159 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.549419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.6.23.194:37301)
0928 14:36:24.549547 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.549552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.549568 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:36:24.549749 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.549751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.549811 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.549917 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:24.550175 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.550689 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.551621 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.551846 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.551957 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.552264 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.552720 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.552849 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.552940 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.552994 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:36:24.556233 14219 raft_consensus.cc:1261] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Refusing update from remote peer ae4e881d45b1478ebb6ffdd0eb67e591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.556344 13941 raft_consensus.cc:1261] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Refusing update from remote peer ae4e881d45b1478ebb6ffdd0eb67e591: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.556862 14337 consensus_queue.cc:1035] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.557063 14302 consensus_queue.cc:1035] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.558763 14303 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: TxnStatusTablet state changed. Reason: New leader ae4e881d45b1478ebb6ffdd0eb67e591. Latest consensus state: current_term: 1 leader_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.558890 14303 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:36:24.559062 14314 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: TxnStatusTablet state changed. Reason: New leader ae4e881d45b1478ebb6ffdd0eb67e591. Latest consensus state: current_term: 1 leader_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.559175 14314 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:36:24.560406 14337 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.560550 14337 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:36:24.560758 14310 mvcc.cc:205] Tried to move back new op lower bound from 7334322931936456704 to 7334322931786989568. Current Snapshot: MvccSnapshot[applied={T|T < 7334322931936456704}]
I20260928 14:36:24.562322 14303 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.562431 14303 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P 1cb8820f933d453fac85ab89fe98a732: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:36:24.564541 14337 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.564704 14337 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P ae4e881d45b1478ebb6ffdd0eb67e591: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:36:24.566903 14314 tablet_replica.cc:439] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } } }
I20260928 14:36:24.567021 14314 tablet_replica.cc:442] T 351b81d34813462998a39d6f2bc0721d P 70a8ed92a63f432eb6ce532d001b55d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:36:24.577091 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.573479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:47045 (local address 127.6.23.194:37301)
0928 14:36:24.573618 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.573623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.573788 (+ 165us) server_negotiation.cc:407] Connection header received
0928 14:36:24.573961 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.573963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.574010 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.574073 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:24.574329 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.574834 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.576095 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.576304 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.576382 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.576698 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.576764 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.576816 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.576935 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.576980 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:36:24.577175 14347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.573351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37301 (local address 127.6.23.193:47045)
0928 14:36:24.573696 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.573712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.573827 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.574106 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.574110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.574118 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:36:24.574231 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.574237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.574939 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.574943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.575984 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.575996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.576415 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.576419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.576509 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.576567 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.576847 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.576853 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.576925 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.577012 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":112,"threads_started":1}
I20260928 14:36:24.583707 14219 raft_consensus.cc:1261] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Refusing update from remote peer 70a8ed92a63f432eb6ce532d001b55d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.583765 14081 raft_consensus.cc:1261] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Refusing update from remote peer 70a8ed92a63f432eb6ce532d001b55d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.584112 14301 consensus_queue.cc:1035] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.584278 14314 consensus_queue.cc:1035] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.613808 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.610250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:59565 (local address 127.6.23.193:38011)
0928 14:36:24.610370 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.610375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.610391 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:36:24.610443 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.610446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.610507 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.610600 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:24.610863 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.611461 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.612399 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.612722 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.612799 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.613261 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.613356 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.613406 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.613581 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.613651 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:36:24.614231 14353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.609886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38011 (local address 127.6.23.194:59565)
0928 14:36:24.610247 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.610263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.610361 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.610607 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.610609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.610618 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:36:24.610757 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.610763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.611575 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.611580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.612235 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.612264 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.612923 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.612926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.613041 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.613098 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.613470 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.613476 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.614002 (+ 526us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.614090 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":118,"threads_started":1}
I20260928 14:36:24.647770 13783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:24.647953 13783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:24.650029 13783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: Sending DeleteTablet for 3 replicas of tablet 5c93927ffaa84f01944710b68c4911a6
I20260928 14:36:24.650585 13921 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c93927ffaa84f01944710b68c4911a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:48688
I20260928 14:36:24.650785 14320 tablet_replica.cc:330] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: stopping tablet replica
I20260928 14:36:24.650794 14201 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c93927ffaa84f01944710b68c4911a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34954
I20260928 14:36:24.650930 14320 raft_consensus.cc:2229] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:36:24.650972 14319 tablet_replica.cc:330] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: stopping tablet replica
I20260928 14:36:24.651082 14319 raft_consensus.cc:2229] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:24.651140 14320 raft_consensus.cc:2258] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.651224 14319 raft_consensus.cc:2258] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.651343 14061 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c93927ffaa84f01944710b68c4911a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:55520
I20260928 14:36:24.651525 14321 tablet_replica.cc:330] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: stopping tablet replica
I20260928 14:36:24.651660 14321 raft_consensus.cc:2229] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:24.651782 14321 raft_consensus.cc:2258] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.652014 14320 ts_tablet_manager.cc:1905] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.652024 14319 ts_tablet_manager.cc:1905] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.654284 14319 ts_tablet_manager.cc:1918] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:36:24.654382 14319 log.cc:1196] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/5c93927ffaa84f01944710b68c4911a6
I20260928 14:36:24.654631 14319 ts_tablet_manager.cc:1939] T 5c93927ffaa84f01944710b68c4911a6 P 1cb8820f933d453fac85ab89fe98a732: Deleting consensus metadata
I20260928 14:36:24.655092 14320 ts_tablet_manager.cc:1918] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:36:24.655187 14320 log.cc:1196] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/5c93927ffaa84f01944710b68c4911a6
I20260928 14:36:24.655489 13771 catalog_manager.cc:4841] TS 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359): tablet 5c93927ffaa84f01944710b68c4911a6 (table test-table [id=2613f7f04b6e4359993f068500f0a327]) successfully deleted
I20260928 14:36:24.655937 14321 ts_tablet_manager.cc:1905] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.658931 14321 ts_tablet_manager.cc:1918] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:36:24.659024 14321 log.cc:1196] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/5c93927ffaa84f01944710b68c4911a6
I20260928 14:36:24.659423 14321 ts_tablet_manager.cc:1939] T 5c93927ffaa84f01944710b68c4911a6 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting consensus metadata
I20260928 14:36:24.659626 14320 ts_tablet_manager.cc:1939] T 5c93927ffaa84f01944710b68c4911a6 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting consensus metadata
I20260928 14:36:24.660261 13770 catalog_manager.cc:4841] TS ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301): tablet 5c93927ffaa84f01944710b68c4911a6 (table test-table [id=2613f7f04b6e4359993f068500f0a327]) successfully deleted
I20260928 14:36:24.660363 13768 catalog_manager.cc:4841] TS 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011): tablet 5c93927ffaa84f01944710b68c4911a6 (table test-table [id=2613f7f04b6e4359993f068500f0a327]) successfully deleted
Sep 28 14:36:24 dist-test-slave-06lh krb5kdc[13739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:36:24 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:24 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:36:24.685696 13833 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:24.677545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.6.23.254:33523)
0928 14:36:24.677716 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.677721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.677912 (+ 191us) server_negotiation.cc:407] Connection header received
0928 14:36:24.678018 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.678022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.678082 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.678192 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:24.678912 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.679835 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.681319 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.681677 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.681781 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.685559 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51910: BlockingRecv error: recv got EOF from 127.0.0.1:51910 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 14:36:24 dist-test-slave-06lh krb5kdc[13739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM
I20260928 14:36:24.701128 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.687813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.6.23.254:33523)
0928 14:36:24.687932 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.687936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.691538 (+ 3602us) server_negotiation.cc:407] Connection header received
0928 14:36:24.691663 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.691666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.691710 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.691778 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:24.692639 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.693227 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.694470 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.694690 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.694772 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.697859 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:24.697863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:24.697864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:24.697889 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:24.699363 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:24.699852 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:24.699878 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:24.699880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:24.699962 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:24.700244 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:24.700247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:24.700249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:24.700448 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:24.700592 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.700830 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.700955 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:36:24.704133 13783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:36:24.704627 13783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:24.709551 13921 tablet_service.cc:1467] Processing CreateTablet for tablet 3154f69812e44b31a588c8032d18feeb (DEFAULT_TABLE table=test-table [id=3c5cf7fdd85a4cd2b6c1b11de0937da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.709550 14061 tablet_service.cc:1467] Processing CreateTablet for tablet 3154f69812e44b31a588c8032d18feeb (DEFAULT_TABLE table=test-table [id=3c5cf7fdd85a4cd2b6c1b11de0937da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.709731 14201 tablet_service.cc:1467] Processing CreateTablet for tablet 3154f69812e44b31a588c8032d18feeb (DEFAULT_TABLE table=test-table [id=3c5cf7fdd85a4cd2b6c1b11de0937da8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.709897 14061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3154f69812e44b31a588c8032d18feeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.709913 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3154f69812e44b31a588c8032d18feeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.710003 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3154f69812e44b31a588c8032d18feeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.712042 14295 tablet_bootstrap.cc:492] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: Bootstrap starting.
I20260928 14:36:24.712352 14297 tablet_bootstrap.cc:492] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: Bootstrap starting.
I20260928 14:36:24.712393 14296 tablet_bootstrap.cc:492] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: Bootstrap starting.
I20260928 14:36:24.712961 14295 tablet_bootstrap.cc:654] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.713275 14297 tablet_bootstrap.cc:654] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.713300 14296 tablet_bootstrap.cc:654] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.714331 14295 tablet_bootstrap.cc:492] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: No bootstrap required, opened a new log
I20260928 14:36:24.714421 14295 ts_tablet_manager.cc:1397] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:24.714458 14297 tablet_bootstrap.cc:492] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: No bootstrap required, opened a new log
I20260928 14:36:24.714469 14296 tablet_bootstrap.cc:492] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: No bootstrap required, opened a new log
I20260928 14:36:24.714522 14297 ts_tablet_manager.cc:1397] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:24.714542 14296 ts_tablet_manager.cc:1397] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:36:24.714880 14296 raft_consensus.cc:348] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.714910 14295 raft_consensus.cc:348] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.715000 14295 raft_consensus.cc:374] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.715000 14296 raft_consensus.cc:374] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.714954 14297 raft_consensus.cc:348] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.715034 14295 raft_consensus.cc:729] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e881d45b1478ebb6ffdd0eb67e591, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.715034 14296 raft_consensus.cc:729] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a8ed92a63f432eb6ce532d001b55d6, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.715061 14297 raft_consensus.cc:374] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.715087 14297 raft_consensus.cc:729] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb8820f933d453fac85ab89fe98a732, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.715161 14296 consensus_queue.cc:260] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.715215 14297 consensus_queue.cc:260] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.715478 14296 ts_tablet_manager.cc:1428] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:24.715507 14297 ts_tablet_manager.cc:1428] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:24.715161 14295 consensus_queue.cc:260] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.716269 14295 ts_tablet_manager.cc:1428] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:24.752169 14303 raft_consensus.cc:482] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:24.752372 14303 raft_consensus.cc:504] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.753592 14303 leader_election.cc:288] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011), ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301)
I20260928 14:36:24.757947 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.753821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37301 (local address 127.6.23.195:38319)
0928 14:36:24.754152 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.754164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.754261 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.754582 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.754587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.754605 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:24.754724 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.754729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.755544 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.755546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.756711 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.756723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.757526 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.757529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.757636 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.757669 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.757731 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.757787 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":85,"threads_started":1}
I20260928 14:36:24.757987 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.753946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:38319 (local address 127.6.23.194:37301)
0928 14:36:24.754119 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.754125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.754243 (+ 118us) server_negotiation.cc:407] Connection header received
0928 14:36:24.754410 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.754413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.754455 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.754552 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:24.754867 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.755429 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.756848 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.757380 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.757529 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.757604 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.757769 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.757839 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 14:36:24.758414 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.753821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38011 (local address 127.6.23.195:40001)
0928 14:36:24.754400 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:24.754411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:24.754506 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:24.754794 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:24.754799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:24.754820 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:24.755334 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.755342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.756154 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.756158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.757129 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:24.757137 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.757979 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.757984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:24.758102 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.758129 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:36:24.758201 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:36:24.758273 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":279,"thread_start_us":128,"threads_started":1}
I20260928 14:36:24.758473 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.753933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:40001 (local address 127.6.23.193:38011)
0928 14:36:24.754077 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.754083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.754509 (+ 426us) server_negotiation.cc:407] Connection header received
0928 14:36:24.754635 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.754641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.754702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.754761 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:24.755459 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.756022 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.757252 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.757848 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.757978 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.758036 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.758232 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.758316 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:36:24.758417 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3154f69812e44b31a588c8032d18feeb" candidate_uuid: "1cb8820f933d453fac85ab89fe98a732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" is_pre_election: true
I20260928 14:36:24.758596 14081 raft_consensus.cc:2454] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb8820f933d453fac85ab89fe98a732 in term 0.
I20260928 14:36:24.758985 14154 leader_election.cc:302] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb8820f933d453fac85ab89fe98a732, ae4e881d45b1478ebb6ffdd0eb67e591; no voters:
I20260928 14:36:24.759114 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3154f69812e44b31a588c8032d18feeb" candidate_uuid: "1cb8820f933d453fac85ab89fe98a732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a8ed92a63f432eb6ce532d001b55d6" is_pre_election: true
I20260928 14:36:24.759306 13941 raft_consensus.cc:2454] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cb8820f933d453fac85ab89fe98a732 in term 0.
I20260928 14:36:24.759356 14303 raft_consensus.cc:2781] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:24.759415 14303 raft_consensus.cc:482] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:24.759471 14303 raft_consensus.cc:3037] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.760547 14303 raft_consensus.cc:504] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.760903 14303 leader_election.cc:288] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [CANDIDATE]: Term 1 election: Requested vote from peers 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011), ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301)
I20260928 14:36:24.761183 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3154f69812e44b31a588c8032d18feeb" candidate_uuid: "1cb8820f933d453fac85ab89fe98a732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70a8ed92a63f432eb6ce532d001b55d6"
I20260928 14:36:24.761330 13941 raft_consensus.cc:3037] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.761291 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3154f69812e44b31a588c8032d18feeb" candidate_uuid: "1cb8820f933d453fac85ab89fe98a732" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591"
I20260928 14:36:24.761425 14081 raft_consensus.cc:3037] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.762279 14081 raft_consensus.cc:2454] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb8820f933d453fac85ab89fe98a732 in term 1.
I20260928 14:36:24.762281 13941 raft_consensus.cc:2454] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cb8820f933d453fac85ab89fe98a732 in term 1.
I20260928 14:36:24.762589 14152 leader_election.cc:302] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cb8820f933d453fac85ab89fe98a732, 70a8ed92a63f432eb6ce532d001b55d6, ae4e881d45b1478ebb6ffdd0eb67e591; no voters:
I20260928 14:36:24.762781 14303 raft_consensus.cc:2781] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:24.763020 14303 raft_consensus.cc:686] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 1 LEADER]: Becoming Leader. State: Replica: 1cb8820f933d453fac85ab89fe98a732, State: Running, Role: LEADER
I20260928 14:36:24.763345 14303 consensus_queue.cc:237] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.765849 13782 catalog_manager.cc:5495] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195). New cstate: current_term: 1 leader_uuid: "1cb8820f933d453fac85ab89fe98a732" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } health_report { overall_health: HEALTHY } } }
I20260928 14:36:24.783982 14294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.780247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.6.23.195:44359)
0928 14:36:24.780403 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.780408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.780424 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:36:24.780604 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.780606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.780662 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.780784 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:24.781054 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.781590 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.782440 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.782674 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.782788 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.783065 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.783591 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.783720 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.783782 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.783835 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:36:24.786633 14081 raft_consensus.cc:1261] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Refusing update from remote peer 1cb8820f933d453fac85ab89fe98a732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.786671 13941 raft_consensus.cc:1261] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Refusing update from remote peer 1cb8820f933d453fac85ab89fe98a732: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.787160 14372 consensus_queue.cc:1035] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.787452 14303 consensus_queue.cc:1035] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.792604 14309 mvcc.cc:205] Tried to move back new op lower bound from 7334322932880994304 to 7334322932793372672. Current Snapshot: MvccSnapshot[applied={T|T < 7334322932880994304 or (T in {7334322932880994304})}]
I20260928 14:36:24.798419 13784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:24.798601 13784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:24.800705 13784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: Sending DeleteTablet for 3 replicas of tablet 3154f69812e44b31a588c8032d18feeb
I20260928 14:36:24.801277 13921 tablet_service.cc:1514] Processing DeleteTablet for tablet 3154f69812e44b31a588c8032d18feeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:48688
I20260928 14:36:24.801314 14061 tablet_service.cc:1514] Processing DeleteTablet for tablet 3154f69812e44b31a588c8032d18feeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:55520
I20260928 14:36:24.801419 14201 tablet_service.cc:1514] Processing DeleteTablet for tablet 3154f69812e44b31a588c8032d18feeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34954
I20260928 14:36:24.801492 14321 tablet_replica.cc:330] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: stopping tablet replica
I20260928 14:36:24.801622 14319 tablet_replica.cc:330] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: stopping tablet replica
I20260928 14:36:24.801620 14321 raft_consensus.cc:2229] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:24.801638 14320 tablet_replica.cc:330] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: stopping tablet replica
I20260928 14:36:24.801723 14319 raft_consensus.cc:2229] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:36:24.801753 14321 raft_consensus.cc:2258] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.801764 14320 raft_consensus.cc:2229] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:24.801893 14320 raft_consensus.cc:2258] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.802016 14319 raft_consensus.cc:2258] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:24.802313 13784 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:36:24.802734 13784 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:24.802789 14320 ts_tablet_manager.cc:1905] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.802801 14319 ts_tablet_manager.cc:1905] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.805107 14320 ts_tablet_manager.cc:1918] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:24.805163 14320 log.cc:1196] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/3154f69812e44b31a588c8032d18feeb
I20260928 14:36:24.805349 14320 ts_tablet_manager.cc:1939] T 3154f69812e44b31a588c8032d18feeb P 70a8ed92a63f432eb6ce532d001b55d6: Deleting consensus metadata
I20260928 14:36:24.805359 14319 ts_tablet_manager.cc:1918] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:24.805429 14319 log.cc:1196] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/3154f69812e44b31a588c8032d18feeb
I20260928 14:36:24.805683 14319 ts_tablet_manager.cc:1939] T 3154f69812e44b31a588c8032d18feeb P 1cb8820f933d453fac85ab89fe98a732: Deleting consensus metadata
I20260928 14:36:24.806141 13768 catalog_manager.cc:4841] TS 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011): tablet 3154f69812e44b31a588c8032d18feeb (table test-table [id=3c5cf7fdd85a4cd2b6c1b11de0937da8]) successfully deleted
I20260928 14:36:24.806447 13771 catalog_manager.cc:4841] TS 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359): tablet 3154f69812e44b31a588c8032d18feeb (table test-table [id=3c5cf7fdd85a4cd2b6c1b11de0937da8]) successfully deleted
I20260928 14:36:24.808092 14321 ts_tablet_manager.cc:1905] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:24.809518 13921 tablet_service.cc:1467] Processing CreateTablet for tablet 56ab427feae4463289128e17dcb78aa8 (DEFAULT_TABLE table=test-table [id=244a7506c6d147808f7beb5cb4c3b7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.809573 14201 tablet_service.cc:1467] Processing CreateTablet for tablet 56ab427feae4463289128e17dcb78aa8 (DEFAULT_TABLE table=test-table [id=244a7506c6d147808f7beb5cb4c3b7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.809818 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ab427feae4463289128e17dcb78aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.809851 14201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ab427feae4463289128e17dcb78aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.810741 14321 ts_tablet_manager.cc:1918] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:24.810818 14321 log.cc:1196] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/3154f69812e44b31a588c8032d18feeb
I20260928 14:36:24.811080 14321 ts_tablet_manager.cc:1939] T 3154f69812e44b31a588c8032d18feeb P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting consensus metadata
I20260928 14:36:24.812219 14297 tablet_bootstrap.cc:492] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: Bootstrap starting.
I20260928 14:36:24.812314 14296 tablet_bootstrap.cc:492] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: Bootstrap starting.
I20260928 14:36:24.813170 14296 tablet_bootstrap.cc:654] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.813269 14297 tablet_bootstrap.cc:654] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.813751 13770 catalog_manager.cc:4841] TS ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301): tablet 3154f69812e44b31a588c8032d18feeb (table test-table [id=3c5cf7fdd85a4cd2b6c1b11de0937da8]) successfully deleted
I20260928 14:36:24.814334 14296 tablet_bootstrap.cc:492] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: No bootstrap required, opened a new log
I20260928 14:36:24.814291 14061 tablet_service.cc:1467] Processing CreateTablet for tablet 56ab427feae4463289128e17dcb78aa8 (DEFAULT_TABLE table=test-table [id=244a7506c6d147808f7beb5cb4c3b7e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:24.814405 14296 ts_tablet_manager.cc:1397] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:24.814560 14061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ab427feae4463289128e17dcb78aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:24.814723 14297 tablet_bootstrap.cc:492] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: No bootstrap required, opened a new log
I20260928 14:36:24.814800 14297 ts_tablet_manager.cc:1397] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:24.814862 14296 raft_consensus.cc:348] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.814996 14296 raft_consensus.cc:374] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.815030 14296 raft_consensus.cc:729] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a8ed92a63f432eb6ce532d001b55d6, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.815145 14296 consensus_queue.cc:260] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.815263 14297 raft_consensus.cc:348] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.815382 14297 raft_consensus.cc:374] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.815418 14297 raft_consensus.cc:729] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb8820f933d453fac85ab89fe98a732, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.815538 14297 consensus_queue.cc:260] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.815718 14296 ts_tablet_manager.cc:1428] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:24.815814 14297 ts_tablet_manager.cc:1428] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:24.817349 14295 tablet_bootstrap.cc:492] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: Bootstrap starting.
I20260928 14:36:24.818228 14295 tablet_bootstrap.cc:654] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:24.819424 14295 tablet_bootstrap.cc:492] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: No bootstrap required, opened a new log
I20260928 14:36:24.819515 14295 ts_tablet_manager.cc:1397] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:24.819898 14295 raft_consensus.cc:348] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.820027 14295 raft_consensus.cc:374] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:24.820060 14295 raft_consensus.cc:729] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4e881d45b1478ebb6ffdd0eb67e591, State: Initialized, Role: FOLLOWER
I20260928 14:36:24.820180 14295 consensus_queue.cc:260] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.820433 14295 ts_tablet_manager.cc:1428] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:36:24.912257 14314 raft_consensus.cc:482] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:24.912432 14314 raft_consensus.cc:504] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.912927 14314 leader_election.cc:288] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301), 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:24.913443 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ab427feae4463289128e17dcb78aa8" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" is_pre_election: true
I20260928 14:36:24.913445 14221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ab427feae4463289128e17dcb78aa8" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb8820f933d453fac85ab89fe98a732" is_pre_election: true
I20260928 14:36:24.913612 14081 raft_consensus.cc:2454] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 0.
I20260928 14:36:24.913641 14221 raft_consensus.cc:2454] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 0.
I20260928 14:36:24.913961 13874 leader_election.cc:302] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70a8ed92a63f432eb6ce532d001b55d6, ae4e881d45b1478ebb6ffdd0eb67e591; no voters:
I20260928 14:36:24.914211 14314 raft_consensus.cc:2781] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:24.914284 14314 raft_consensus.cc:482] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:24.914341 14314 raft_consensus.cc:3037] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.915555 14314 raft_consensus.cc:504] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.916013 14314 leader_election.cc:288] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 election: Requested vote from peers ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301), 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359)
I20260928 14:36:24.916412 14081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ab427feae4463289128e17dcb78aa8" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591"
I20260928 14:36:24.916411 14221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ab427feae4463289128e17dcb78aa8" candidate_uuid: "70a8ed92a63f432eb6ce532d001b55d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cb8820f933d453fac85ab89fe98a732"
I20260928 14:36:24.916544 14221 raft_consensus.cc:3037] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.916544 14081 raft_consensus.cc:3037] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:24.917703 14221 raft_consensus.cc:2454] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 1.
I20260928 14:36:24.917703 14081 raft_consensus.cc:2454] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70a8ed92a63f432eb6ce532d001b55d6 in term 1.
I20260928 14:36:24.918097 13875 leader_election.cc:302] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cb8820f933d453fac85ab89fe98a732, 70a8ed92a63f432eb6ce532d001b55d6; no voters:
I20260928 14:36:24.918284 14314 raft_consensus.cc:2781] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:24.918394 14314 raft_consensus.cc:686] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 LEADER]: Becoming Leader. State: Replica: 70a8ed92a63f432eb6ce532d001b55d6, State: Running, Role: LEADER
I20260928 14:36:24.918553 14314 consensus_queue.cc:237] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } }
I20260928 14:36:24.920360 13784 catalog_manager.cc:5495] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "70a8ed92a63f432eb6ce532d001b55d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70a8ed92a63f432eb6ce532d001b55d6" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 38011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 } health_report { overall_health: UNKNOWN } } }
I20260928 14:36:24.957747 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:24.953979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.6.23.193:38011)
0928 14:36:24.954095 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:24.954098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:24.954129 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:36:24.954282 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:24.954284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:24.954327 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:24.954392 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:24.954671 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.955170 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.956440 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:24.956797 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:24.956920 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:24.957199 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:24.957303 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:24.957380 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:24.957544 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 14:36:24.957600 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:36:24.964540 14081 raft_consensus.cc:1261] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Refusing update from remote peer 70a8ed92a63f432eb6ce532d001b55d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.964656 14221 raft_consensus.cc:1261] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Refusing update from remote peer 70a8ed92a63f432eb6ce532d001b55d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:24.965005 14306 consensus_queue.cc:1035] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cb8820f933d453fac85ab89fe98a732" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.965302 14314 consensus_queue.cc:1035] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae4e881d45b1478ebb6ffdd0eb67e591" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:24.998479 13783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:24.998625 13783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:25.000542 13783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1bf86f0785b4cbd9efbad35f48695b2: Sending DeleteTablet for 3 replicas of tablet 56ab427feae4463289128e17dcb78aa8
I20260928 14:36:25.001057 14061 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ab427feae4463289128e17dcb78aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:55520
I20260928 14:36:25.001053 13921 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ab427feae4463289128e17dcb78aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:48688
I20260928 14:36:25.001175 14201 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ab427feae4463289128e17dcb78aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:24 UTC) from {username='oryx', principal='oryx/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34954
I20260928 14:36:25.001230 14321 tablet_replica.cc:330] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: stopping tablet replica
I20260928 14:36:25.001230 14320 tablet_replica.cc:330] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: stopping tablet replica
I20260928 14:36:25.001331 14320 raft_consensus.cc:2229] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:36:25.001335 14321 raft_consensus.cc:2229] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:25.001350 14319 tablet_replica.cc:330] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: stopping tablet replica
I20260928 14:36:25.001441 14321 raft_consensus.cc:2258] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:25.001489 14319 raft_consensus.cc:2229] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:25.001542 14320 raft_consensus.cc:2258] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:25.001607 14319 raft_consensus.cc:2258] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:25.002255 14321 ts_tablet_manager.cc:1905] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:25.002403 14319 ts_tablet_manager.cc:1905] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:25.002614 14320 ts_tablet_manager.cc:1905] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:25.004487 14321 ts_tablet_manager.cc:1918] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:36:25.004580 14321 log.cc:1196] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/56ab427feae4463289128e17dcb78aa8
I20260928 14:36:25.004818 14321 ts_tablet_manager.cc:1939] T 56ab427feae4463289128e17dcb78aa8 P ae4e881d45b1478ebb6ffdd0eb67e591: Deleting consensus metadata
I20260928 14:36:25.004908 14319 ts_tablet_manager.cc:1918] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:36:25.004907 14320 ts_tablet_manager.cc:1918] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:36:25.004995 14320 log.cc:1196] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/56ab427feae4463289128e17dcb78aa8
I20260928 14:36:25.005007 14319 log.cc:1196] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/56ab427feae4463289128e17dcb78aa8
I20260928 14:36:25.005261 14320 ts_tablet_manager.cc:1939] T 56ab427feae4463289128e17dcb78aa8 P 70a8ed92a63f432eb6ce532d001b55d6: Deleting consensus metadata
I20260928 14:36:25.005261 14319 ts_tablet_manager.cc:1939] T 56ab427feae4463289128e17dcb78aa8 P 1cb8820f933d453fac85ab89fe98a732: Deleting consensus metadata
I20260928 14:36:25.005591 13770 catalog_manager.cc:4841] TS ae4e881d45b1478ebb6ffdd0eb67e591 (127.6.23.194:37301): tablet 56ab427feae4463289128e17dcb78aa8 (table test-table [id=244a7506c6d147808f7beb5cb4c3b7e5]) successfully deleted
I20260928 14:36:25.005695 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 13856
I20260928 14:36:25.008531 13771 catalog_manager.cc:4841] TS 1cb8820f933d453fac85ab89fe98a732 (127.6.23.195:44359): tablet 56ab427feae4463289128e17dcb78aa8 (table test-table [id=244a7506c6d147808f7beb5cb4c3b7e5]) successfully deleted
W20260928 14:36:25.014073 13768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:38011 (error 108)
I20260928 14:36:25.014367 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 13996
I20260928 14:36:25.016626 14386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:25.015835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:38011 (local address 127.0.0.1:48702)
0928 14:36:25.016434 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:25.016499 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:38011: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":57,"threads_started":1}
W20260928 14:36:25.016757 13768 catalog_manager.cc:4584] TS 70a8ed92a63f432eb6ce532d001b55d6 (127.6.23.193:38011): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56ab427feae4463289128e17dcb78aa8: Network error: Client connection negotiation failed: client connection to 127.6.23.193:38011: connect: Connection refused (error 111)
I20260928 14:36:25.021744 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 14136
I20260928 14:36:25.028856 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 13752
2026-09-28T14:36:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3290 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:25 dist-test-slave-06lh krb5kdc[14389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:25 dist-test-slave-06lh krb5kdc[14389](info): set up 2 sockets
Sep 28 14:36:25 dist-test-slave-06lh krb5kdc[14389](info): commencing operation
krb5kdc: starting...
W20260928 14:36:26.064455 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:26 dist-test-slave-06lh krb5kdc[14389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:36:26.188130 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:46235
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:45187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:46235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:26.301244 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:26.301530 14402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:26.301571 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:26.305557 14402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:26.305614 14402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:26.305634 14402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:26.305650 14402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:26.305665 14402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:26.310952 14402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:45187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:46235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:46235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:26.312121 14402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:26.313120 14402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:26.322184 14408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:26.322198 14410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:26.322233 14407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:26.322958 14402 server_base.cc:1034] running on GCE node
I20260928 14:36:26.323673 14402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:26.324832 14402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:26.326031 14402 hybrid_clock.cc:648] HybridClock initialized: now 1790606186326011 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T14:36:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1305 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:26 dist-test-slave-06lh krb5kdc[14415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:26 dist-test-slave-06lh krb5kdc[14415](info): set up 2 sockets
Sep 28 14:36:26 dist-test-slave-06lh krb5kdc[14415](info): commencing operation
krb5kdc: starting...
W20260928 14:36:27.376271 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:27 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:36:27.500046 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39511
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:39511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:27.612947 14428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:27.613287 14428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:27.613335 14428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:27.617436 14428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:27.617514 14428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:27.617535 14428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:27.617552 14428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:27.617568 14428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:27.623082 14428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:39511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:27.624244 14428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:27.625267 14428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:27.634176 14433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:27.634238 14428 server_base.cc:1034] running on GCE node
W20260928 14:36:27.634177 14436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:27.634207 14434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:27.635044 14428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:27.636482 14428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:27.637667 14428 hybrid_clock.cc:648] HybridClock initialized: now 1790606187637631 us; error 49 us; skew 500 ppm
Sep 28 14:36:27 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606187, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:27.640666 14428 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:36:27.641672 14428 webserver.cc:460] Webserver started at http://127.6.23.254:36969/ using document root <none> and password file <none>
I20260928 14:36:27.642249 14428 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:27.642310 14428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:27.642550 14428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:27.644723 14428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "46b167e18df84ca4948f3ac73032d9e5"
format_stamp: "Formatted at 2026-09-28 14:36:27 on dist-test-slave-06lh"
server_key: "a7a3613e123740151784cb2c9305e44f"
server_key_iv: "422346cd5947440a0a3dda3954cc4408"
server_key_version: "encryptionkey@0"
I20260928 14:36:27.645323 14428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "46b167e18df84ca4948f3ac73032d9e5"
format_stamp: "Formatted at 2026-09-28 14:36:27 on dist-test-slave-06lh"
server_key: "a7a3613e123740151784cb2c9305e44f"
server_key_iv: "422346cd5947440a0a3dda3954cc4408"
server_key_version: "encryptionkey@0"
I20260928 14:36:27.649235 14428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:27.651789 14443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:27.653025 14428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:27.653168 14428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "46b167e18df84ca4948f3ac73032d9e5"
format_stamp: "Formatted at 2026-09-28 14:36:27 on dist-test-slave-06lh"
server_key: "a7a3613e123740151784cb2c9305e44f"
server_key_iv: "422346cd5947440a0a3dda3954cc4408"
server_key_version: "encryptionkey@0"
I20260928 14:36:27.653285 14428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:27.677732 14428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:27.681311 14428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:27.681540 14428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:27.689333 14428 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:39511
I20260928 14:36:27.689375 14495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:39511 every 8 connection(s)
I20260928 14:36:27.690640 14428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:27.693638 14496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:27.696517 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 14428
I20260928 14:36:27.696662 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:27.696982 6239 external_mini_cluster.cc:1428] Setting key 8d894b14381d6a3f3daee106b92fce65
I20260928 14:36:27.700234 14496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Bootstrap starting.
I20260928 14:36:27.703058 14496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:27.703992 14496 log.cc:824] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:27.706091 14496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: No bootstrap required, opened a new log
Sep 28 14:36:27 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:27.710342 14496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } }
I20260928 14:36:27.710649 14496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:27.710711 14496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b167e18df84ca4948f3ac73032d9e5, State: Initialized, Role: FOLLOWER
I20260928 14:36:27.711308 14496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } }
I20260928 14:36:27.711447 14496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:27.711529 14496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:27.711627 14496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:27.712159 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:27.698428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50036 (local address 127.6.23.254:39511)
0928 14:36:27.698880 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:27.698897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:27.698940 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:36:27.699654 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:27.699675 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:27.700044 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:27.700366 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:27.701321 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:27.702332 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:27.703573 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:27.704097 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:27.704299 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:27.707050 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:27.707056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:27.707070 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:27.707098 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:27.708908 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:27.709413 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:27.709419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:27.709425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:27.709540 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:27.709823 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:27.709825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:27.709826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:27.710234 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:27.710390 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:27.710766 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 14:36:27.711135 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":142,"threads_started":1}
I20260928 14:36:27.712677 14496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } }
I20260928 14:36:27.713029 14496 leader_election.cc:302] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b167e18df84ca4948f3ac73032d9e5; no voters:
I20260928 14:36:27.713389 14496 leader_election.cc:288] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:27.713524 14501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:27.713704 14501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 1 LEADER]: Becoming Leader. State: Replica: 46b167e18df84ca4948f3ac73032d9e5, State: Running, Role: LEADER
I20260928 14:36:27.713999 14501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } }
I20260928 14:36:27.714701 14496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:27.715214 14501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b167e18df84ca4948f3ac73032d9e5. Latest consensus state: current_term: 1 leader_uuid: "46b167e18df84ca4948f3ac73032d9e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } } }
I20260928 14:36:27.715350 14501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:27.715873 14502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46b167e18df84ca4948f3ac73032d9e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } } }
I20260928 14:36:27.715972 14502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:27.716295 14505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:27.719750 14505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:27.725205 14505 catalog_manager.cc:1347] Generated new cluster ID: 207a42f69eb4411bb56948a542ace93b
I20260928 14:36:27.725281 14505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:27.736634 14505 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:27.737676 14505 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:27.745443 14505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Generated new TSK 0
I20260928 14:36:27.746224 14505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:36:27.776803 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39511
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:27.905805 14522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:27.906080 14522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:27.906123 14522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:27.910213 14522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:27.910280 14522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:27.910382 14522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:27.915827 14522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:27.916932 14522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:27.917982 14522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:27.927759 14530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:27.927750 14528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:27.927800 14527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:27.928120 14522 server_base.cc:1034] running on GCE node
I20260928 14:36:27.928570 14522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:27.929186 14522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:27.930346 14522 hybrid_clock.cc:648] HybridClock initialized: now 1790606187930321 us; error 37 us; skew 500 ppm
Sep 28 14:36:27 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606187, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:27.933346 14522 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:36:27.934288 14522 webserver.cc:460] Webserver started at http://127.6.23.193:44731/ using document root <none> and password file <none>
I20260928 14:36:27.934825 14522 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:27.934886 14522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:27.935127 14522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:27.937273 14522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "e863944a8c06456b8ae44869e06645b3"
format_stamp: "Formatted at 2026-09-28 14:36:27 on dist-test-slave-06lh"
server_key: "dfe6e65545cbaf3319e3b0bbe64b4004"
server_key_iv: "9f34a0e0222398a3aeb7dba2e542f151"
server_key_version: "encryptionkey@0"
I20260928 14:36:27.937866 14522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "e863944a8c06456b8ae44869e06645b3"
format_stamp: "Formatted at 2026-09-28 14:36:27 on dist-test-slave-06lh"
server_key: "dfe6e65545cbaf3319e3b0bbe64b4004"
server_key_iv: "9f34a0e0222398a3aeb7dba2e542f151"
server_key_version: "encryptionkey@0"
I20260928 14:36:27.941740 14522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:36:27.944268 14537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:27.945528 14522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:27.945657 14522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "e863944a8c06456b8ae44869e06645b3"
format_stamp: "Formatted at 2026-09-28 14:36:27 on dist-test-slave-06lh"
server_key: "dfe6e65545cbaf3319e3b0bbe64b4004"
server_key_iv: "9f34a0e0222398a3aeb7dba2e542f151"
server_key_version: "encryptionkey@0"
I20260928 14:36:27.945773 14522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:27.992890 14522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:27.996302 14522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:27.996524 14522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:27.997143 14522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:27.998001 14522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:27.998047 14522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:27.998121 14522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:27.998152 14522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.007018 14522 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:35515
I20260928 14:36:28.007091 14650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:35515 every 8 connection(s)
I20260928 14:36:28.008025 14522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:28.014418 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 14522
I20260928 14:36:28.014598 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:28.014930 6239 external_mini_cluster.cc:1428] Setting key f5cccc7f6fe1851933c99a91cc616a2e
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606187, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:28.023878 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.009929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:35451 (local address 127.6.23.254:39511)
0928 14:36:28.010067 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:28.010072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:28.010909 (+ 837us) server_negotiation.cc:407] Connection header received
0928 14:36:28.012222 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:28.012225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:28.012274 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:28.012356 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:28.013611 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.014163 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.015941 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.016282 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.016402 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.019844 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:28.019847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:28.019849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:28.019876 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:28.021536 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:28.022156 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:28.022160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:28.022162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:28.022247 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:28.022643 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:28.022646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:28.022647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:28.022840 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:28.022964 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:28.023571 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 14:36:28.023709 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:36:28.024791 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.010257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:35451)
0928 14:36:28.010717 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:28.010762 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:28.012026 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:28.012589 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:28.012596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:28.012956 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:28.013447 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.013461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.014306 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.014310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.015816 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.015833 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.016450 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.016456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.016659 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.017314 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:28.017361 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:28.019630 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:28.021687 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:28.021695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:28.021716 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:28.022039 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:28.022351 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:28.022354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:28.022356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:28.022535 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:28.022961 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:28.022969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:28.023296 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 14:36:28.023506 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 14:36:28.023881 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":134,"threads_started":1}
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:36:28.026208 14651 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39511
I20260928 14:36:28.026559 14651 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:28.027163 14651 heartbeater.cc:511] Master 127.6.23.254:39511 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:36:28.028940 14460 ts_manager.cc:194] Registered new tserver with Master: e863944a8c06456b8ae44869e06645b3 (127.6.23.193:35515)
I20260928 14:36:28.030148 14460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:35451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:36:28.046085 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39511
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:28.158471 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:28.158759 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:28.158804 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:28.162951 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:28.163022 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:28.163085 14656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:28.168722 14656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:28.169754 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:28.170745 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:28.180438 14664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:28.180590 14656 server_base.cc:1034] running on GCE node
W20260928 14:36:28.180449 14661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:28.180826 14662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:28.181277 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:28.181977 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:28.183156 14656 hybrid_clock.cc:648] HybridClock initialized: now 1790606188183143 us; error 34 us; skew 500 ppm
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606188, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:28.191180 14656 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:36:28.192152 14656 webserver.cc:460] Webserver started at http://127.6.23.194:38173/ using document root <none> and password file <none>
I20260928 14:36:28.192737 14656 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:28.192799 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:28.193054 14656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:28.195189 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "98cfa520b88c45f296af35796a4dbd08"
format_stamp: "Formatted at 2026-09-28 14:36:28 on dist-test-slave-06lh"
server_key: "caa4062842cc366bb33cef33d81a3443"
server_key_iv: "21a65aed422086990603faf6eda943f9"
server_key_version: "encryptionkey@0"
I20260928 14:36:28.195811 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "98cfa520b88c45f296af35796a4dbd08"
format_stamp: "Formatted at 2026-09-28 14:36:28 on dist-test-slave-06lh"
server_key: "caa4062842cc366bb33cef33d81a3443"
server_key_iv: "21a65aed422086990603faf6eda943f9"
server_key_version: "encryptionkey@0"
I20260928 14:36:28.199623 14656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:36:28.202136 14671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.203528 14656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:28.203625 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "98cfa520b88c45f296af35796a4dbd08"
format_stamp: "Formatted at 2026-09-28 14:36:28 on dist-test-slave-06lh"
server_key: "caa4062842cc366bb33cef33d81a3443"
server_key_iv: "21a65aed422086990603faf6eda943f9"
server_key_version: "encryptionkey@0"
I20260928 14:36:28.203759 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:28.218504 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:28.221705 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:28.221925 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:28.222599 14656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:28.223654 14656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:28.223703 14656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.223767 14656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:28.223799 14656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.232915 14656 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:46735
I20260928 14:36:28.232942 14784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:46735 every 8 connection(s)
I20260928 14:36:28.233852 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:28.242980 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 14656
I20260928 14:36:28.243108 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606188, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:28.243458 6239 external_mini_cluster.cc:1428] Setting key e08e2c0268e61c419916c519f2301e69
I20260928 14:36:28.248070 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.235769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:44779 (local address 127.6.23.254:39511)
0928 14:36:28.235944 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:28.235950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:28.236793 (+ 843us) server_negotiation.cc:407] Connection header received
0928 14:36:28.238065 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:28.238068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:28.238116 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:28.238191 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:28.239404 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.239925 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.241046 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.241267 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.241348 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.244107 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:28.244117 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:28.244120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:28.244153 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:28.245843 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:28.246397 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:28.246403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:28.246408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:28.246485 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:28.246840 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:28.246851 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:28.246853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:28.247042 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:28.247152 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:28.247759 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 14:36:28.247900 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:36:28.249033 14787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.236183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:44779)
0928 14:36:28.236640 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:28.236683 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:28.237858 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:28.238354 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:28.238360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:28.238719 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:28.239208 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.239280 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.240033 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.240036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.240933 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.240942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.241398 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.241401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.241544 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.242113 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:28.242129 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:28.243933 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:28.245970 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:28.245977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:28.245994 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:28.246270 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:28.246583 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:28.246586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:28.246588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:28.246744 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:28.247152 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:28.247160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:28.247507 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 14:36:28.247707 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 14:36:28.248066 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":127,"threads_started":1}
I20260928 14:36:28.250270 14785 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39511
I20260928 14:36:28.250572 14785 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:28.251273 14785 heartbeater.cc:511] Master 127.6.23.254:39511 requested a full tablet report, sending...
I20260928 14:36:28.252456 14460 ts_manager.cc:194] Registered new tserver with Master: 98cfa520b88c45f296af35796a4dbd08 (127.6.23.194:46735)
I20260928 14:36:28.253055 14460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:44779
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:36:28.272307 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39511
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:28.386818 14790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:28.387109 14790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:28.387153 14790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:28.391333 14790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:28.391400 14790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:28.391492 14790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:28.397042 14790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:39511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:28.398181 14790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:28.399302 14790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:28.409643 14796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:28.409632 14795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:28.409632 14798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:28.409816 14790 server_base.cc:1034] running on GCE node
I20260928 14:36:28.410377 14790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:28.411056 14790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:28.412276 14790 hybrid_clock.cc:648] HybridClock initialized: now 1790606188412260 us; error 34 us; skew 500 ppm
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606188, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:28.415313 14790 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:36:28.416389 14790 webserver.cc:460] Webserver started at http://127.6.23.195:43681/ using document root <none> and password file <none>
I20260928 14:36:28.416972 14790 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:28.417035 14790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:28.417236 14790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:28.419371 14790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "5c7c8dc80da24357a2a4ba057b6440fe"
format_stamp: "Formatted at 2026-09-28 14:36:28 on dist-test-slave-06lh"
server_key: "a71e1804beb8ac274772b19c2ce5de22"
server_key_iv: "f4a852c6808f1fee8a2f97f203b94c84"
server_key_version: "encryptionkey@0"
I20260928 14:36:28.419906 14790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "5c7c8dc80da24357a2a4ba057b6440fe"
format_stamp: "Formatted at 2026-09-28 14:36:28 on dist-test-slave-06lh"
server_key: "a71e1804beb8ac274772b19c2ce5de22"
server_key_iv: "f4a852c6808f1fee8a2f97f203b94c84"
server_key_version: "encryptionkey@0"
I20260928 14:36:28.423790 14790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:36:28.426406 14805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.427907 14790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:28.428040 14790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "5c7c8dc80da24357a2a4ba057b6440fe"
format_stamp: "Formatted at 2026-09-28 14:36:28 on dist-test-slave-06lh"
server_key: "a71e1804beb8ac274772b19c2ce5de22"
server_key_iv: "f4a852c6808f1fee8a2f97f203b94c84"
server_key_version: "encryptionkey@0"
I20260928 14:36:28.428160 14790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:28.457938 14790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:28.461225 14790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:28.461454 14790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:28.462100 14790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:28.462978 14790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:28.463030 14790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.463101 14790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:28.463133 14790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.472762 14790 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:40677
I20260928 14:36:28.472790 14918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:40677 every 8 connection(s)
I20260928 14:36:28.473712 14790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:28.479382 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 14790
I20260928 14:36:28.479498 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:28.479799 6239 external_mini_cluster.cc:1428] Setting key 8d34322e9492860d6d589bb606cff408
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606188, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:28.488405 14499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.475661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:57405 (local address 127.6.23.254:39511)
0928 14:36:28.475802 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:28.475811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:28.476810 (+ 999us) server_negotiation.cc:407] Connection header received
0928 14:36:28.478205 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:28.478208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:28.478259 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:28.478340 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:28.479805 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.480401 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.481959 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.482204 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.482288 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.484980 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:28.484984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:28.484986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:28.485012 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:28.486322 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:28.486833 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:28.486836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:28.486837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:28.486922 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:28.487290 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:28.487294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:28.487297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:28.487544 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:28.487627 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:28.488143 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 14:36:28.488249 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:36:28.489249 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.476064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:57405)
0928 14:36:28.476637 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:28.476681 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:28.477963 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:28.478543 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:28.478550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:28.478920 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:28.479654 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.479671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.480553 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.480556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.481829 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.481842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.482340 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.482343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.482501 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.483131 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:28.483148 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:28.484783 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:28.486436 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:28.486441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:28.486454 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:28.486716 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:28.487013 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:28.487016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:28.487017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:28.487151 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:28.487647 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:28.487652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:28.487923 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 14:36:28.488093 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 14:36:28.488357 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":120,"threads_started":1}
I20260928 14:36:28.490542 14919 heartbeater.cc:347] Connected to a master server at 127.6.23.254:39511
I20260928 14:36:28.490760 14919 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:28.491214 14919 heartbeater.cc:511] Master 127.6.23.254:39511 requested a full tablet report, sending...
I20260928 14:36:28.492378 14460 ts_manager.cc:194] Registered new tserver with Master: 5c7c8dc80da24357a2a4ba057b6440fe (127.6.23.195:40677)
I20260928 14:36:28.493145 14460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:57405
I20260928 14:36:28.493969 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab.
Entry for principal oryx/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab.
WARNING: no policy specified for oryx/127.6.23.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab.
Entry for principal oryx/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab.
WARNING: no policy specified for oryx/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab.
Entry for principal oryx/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab.
I20260928 14:36:28.570817 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 14522
I20260928 14:36:28.576947 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 14656
I20260928 14:36:28.582511 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 14790
I20260928 14:36:28.589547 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 14428
I20260928 14:36:28.596050 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39511
--webserver_interface=127.6.23.254
--webserver_port=36969
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:39511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:28.713565 14928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:28.713891 14928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:28.713973 14928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:28.718087 14928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:28.718189 14928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:28.718230 14928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:28.718269 14928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:28.718304 14928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:28.723927 14928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:39511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:28.725071 14928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:28.726179 14928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:28.734879 14933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:28.734889 14936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:28.735056 14928 server_base.cc:1034] running on GCE node
W20260928 14:36:28.734901 14934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:28.735857 14928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:28.736980 14928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:28.738142 14928 hybrid_clock.cc:648] HybridClock initialized: now 1790606188738126 us; error 28 us; skew 500 ppm
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606188, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:28.741000 14928 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:36:28.741869 14928 webserver.cc:460] Webserver started at http://127.6.23.254:36969/ using document root <none> and password file <none>
I20260928 14:36:28.742413 14928 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:28.742486 14928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:28.746373 14928 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:28.748534 14943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.749780 14928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:28.749936 14928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "46b167e18df84ca4948f3ac73032d9e5"
format_stamp: "Formatted at 2026-09-28 14:36:27 on dist-test-slave-06lh"
server_key: "a7a3613e123740151784cb2c9305e44f"
server_key_iv: "422346cd5947440a0a3dda3954cc4408"
server_key_version: "encryptionkey@0"
I20260928 14:36:28.750501 14928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:28.765249 14928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:28.768693 14928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:28.768909 14928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:28.776038 14928 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:39511
I20260928 14:36:28.776065 14995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:39511 every 8 connection(s)
I20260928 14:36:28.777110 14928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:28.781244 14996 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:36:28.782578 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 14928
I20260928 14:36:28.782915 14996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Bootstrap starting.
I20260928 14:36:28.783048 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:35515
--local_ip_for_outbound_sockets=127.6.23.193
--tserver_master_addrs=127.6.23.254:39511
--webserver_port=44731
--webserver_interface=127.6.23.193
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.23.193
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:36:28.799823 14996 log.cc:824] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:28.806543 14996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:36:28.806980 14996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Bootstrap complete.
I20260928 14:36:28.811393 14996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } }
I20260928 14:36:28.811841 14996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b167e18df84ca4948f3ac73032d9e5, State: Initialized, Role: FOLLOWER
I20260928 14:36:28.812319 14996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } }
I20260928 14:36:28.812469 14996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:28.812553 14996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:28.812666 14996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:36:28.814615 14996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } }
I20260928 14:36:28.814952 14996 leader_election.cc:302] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b167e18df84ca4948f3ac73032d9e5; no voters:
I20260928 14:36:28.815299 14996 leader_election.cc:288] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:36:28.815378 15000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:36:28.815567 15000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [term 2 LEADER]: Becoming Leader. State: Replica: 46b167e18df84ca4948f3ac73032d9e5, State: Running, Role: LEADER
I20260928 14:36:28.815856 15000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } }
I20260928 14:36:28.816545 14996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:28.817178 15001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "46b167e18df84ca4948f3ac73032d9e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } } }
I20260928 14:36:28.817307 15001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:28.818248 15006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:28.818419 15002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b167e18df84ca4948f3ac73032d9e5. Latest consensus state: current_term: 2 leader_uuid: "46b167e18df84ca4948f3ac73032d9e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b167e18df84ca4948f3ac73032d9e5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39511 } } }
I20260928 14:36:28.818595 15002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:28.821568 15006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:28.822220 15006 catalog_manager.cc:1259] Loaded cluster ID: 207a42f69eb4411bb56948a542ace93b
I20260928 14:36:28.822283 15006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:28.824625 15006 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:28.825284 15006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 46b167e18df84ca4948f3ac73032d9e5: Loaded TSK: 0
I20260928 14:36:28.826498 15006 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:36:28.899669 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:28.899971 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:28.900067 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:28.904165 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:28.904258 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:28.904347 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:28.909838 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.193.keytab
--principal=oryx/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:35515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=44731
--tserver_master_addrs=127.6.23.254:39511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:28.910938 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:28.912081 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:28.922037 15022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:28.922089 15025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:28.922060 15023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:28.922513 14998 server_base.cc:1034] running on GCE node
I20260928 14:36:28.922978 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:28.923838 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:28.925055 14998 hybrid_clock.cc:648] HybridClock initialized: now 1790606188925030 us; error 32 us; skew 500 ppm
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606188, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:28.928629 14998 init.cc:377] Logged in from keytab as oryx/127.6.23.193@KRBTEST.COM (short username oryx)
I20260928 14:36:28.929800 14998 webserver.cc:460] Webserver started at http://127.6.23.193:44731/ using document root <none> and password file <none>
I20260928 14:36:28.930533 14998 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:28.930646 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:28.935920 14998 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:36:28.938705 15032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.940420 14998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:28.940551 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "e863944a8c06456b8ae44869e06645b3"
format_stamp: "Formatted at 2026-09-28 14:36:27 on dist-test-slave-06lh"
server_key: "dfe6e65545cbaf3319e3b0bbe64b4004"
server_key_iv: "9f34a0e0222398a3aeb7dba2e542f151"
server_key_version: "encryptionkey@0"
I20260928 14:36:28.941289 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:28.957350 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:28.961078 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:28.961298 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:28.962038 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:28.963193 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:28.963294 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.963353 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:28.963385 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:28.974272 14998 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:35515
I20260928 14:36:28.974314 15145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:35515 every 8 connection(s)
I20260928 14:36:28.975674 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:28.979653 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 14998
I20260928 14:36:28.980131 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:46735
--local_ip_for_outbound_sockets=127.6.23.194
--tserver_master_addrs=127.6.23.254:39511
--webserver_port=38173
--webserver_interface=127.6.23.194
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.23.194
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:28.991860 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.978238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:58403)
0928 14:36:28.978964 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:28.979017 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:28.980444 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:28.981857 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:28.981868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:28.982398 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:28.983102 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.983120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.985191 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.985194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.986099 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.986110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.987390 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.987393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.987554 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.988233 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:28.988256 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:28.990971 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":166,"threads_started":1}
W20260928 14:36:28.992316 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:28.978140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:58403 (local address 127.6.23.254:39511)
0928 14:36:28.978650 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:28.978670 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:28.979156 (+ 486us) server_negotiation.cc:407] Connection header received
0928 14:36:28.980906 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:28.980924 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:28.981330 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:28.981726 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:28.983318 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.985031 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.986240 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.987128 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.987365 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.991354 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:58403: BlockingRecv error: recv got EOF from 127.6.23.193:58403 (error 108)
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":120,"threads_started":1}
W20260928 14:36:28.992647 15146 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:39511 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.23.254:39511: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:28.998708 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.993135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:49581)
0928 14:36:28.993246 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:28.993265 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:28.993340 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:28.993589 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:28.993591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:28.993729 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:28.993840 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.993844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.994854 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.994857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.995498 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:28.995507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.996104 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.996107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:28.996182 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.996683 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:28.996695 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:28.998605 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 14:36:28.998900 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:28.993218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:49581 (local address 127.6.23.254:39511)
0928 14:36:28.993327 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:28.993333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:28.993348 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:28.993419 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:28.993422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:28.993473 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:28.993557 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:28.993942 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.994742 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.995610 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:28.995966 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:28.996064 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:28.998774 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:49581: BlockingRecv error: recv got EOF from 127.6.23.193:49581 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:28 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:29.004561 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:28.999389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:44895)
0928 14:36:28.999513 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:28.999529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:28.999613 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:28.999838 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:28.999840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:28.999974 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:29.000043 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.000047 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.001045 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.001048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.001601 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.001608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.002154 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.002157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.002212 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.002676 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:29.002687 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:29.004465 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:36:29.004734 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:28.999480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:44895 (local address 127.6.23.254:39511)
0928 14:36:28.999588 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:28.999592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:28.999606 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:36:28.999695 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:28.999698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:28.999740 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:28.999816 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:29.000139 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.000935 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.001707 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.002054 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.002151 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.004618 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:44895: BlockingRecv error: recv got EOF from 127.6.23.193:44895 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 14:36:29.004765 15146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:36:29.098233 15150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:29.098500 15150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:29.098541 15150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:29.102500 15150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:29.102559 15150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:29.102615 15150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:29.107806 15150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.194.keytab
--principal=oryx/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:46735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=38173
--tserver_master_addrs=127.6.23.254:39511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:29.108814 15150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:29.109779 15150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:29.119926 15156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:29.119926 15155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:29.119969 15158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:29.120218 15150 server_base.cc:1034] running on GCE node
I20260928 14:36:29.120599 15150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:29.121245 15150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:29.122412 15150 hybrid_clock.cc:648] HybridClock initialized: now 1790606189122383 us; error 38 us; skew 500 ppm
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606189, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:29.125322 15150 init.cc:377] Logged in from keytab as oryx/127.6.23.194@KRBTEST.COM (short username oryx)
I20260928 14:36:29.126331 15150 webserver.cc:460] Webserver started at http://127.6.23.194:38173/ using document root <none> and password file <none>
I20260928 14:36:29.126991 15150 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:29.127091 15150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:29.131218 15150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:29.133404 15165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:29.134791 15150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:29.134953 15150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "98cfa520b88c45f296af35796a4dbd08"
format_stamp: "Formatted at 2026-09-28 14:36:28 on dist-test-slave-06lh"
server_key: "caa4062842cc366bb33cef33d81a3443"
server_key_iv: "21a65aed422086990603faf6eda943f9"
server_key_version: "encryptionkey@0"
I20260928 14:36:29.135596 15150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:29.152918 15150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:29.156141 15150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:29.156345 15150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:29.156945 15150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:29.157826 15150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:29.157900 15150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:29.157969 15150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:29.158015 15150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:29.167070 15150 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:46735
I20260928 14:36:29.167096 15278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:46735 every 8 connection(s)
I20260928 14:36:29.168207 15150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:29.177929 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 15150
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:29.178467 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:40677
--local_ip_for_outbound_sockets=127.6.23.195
--tserver_master_addrs=127.6.23.254:39511
--webserver_port=43681
--webserver_interface=127.6.23.195
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.23.195
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:29.179668 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:29.170092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:49559 (local address 127.6.23.254:39511)
0928 14:36:29.170232 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:29.170236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:29.171107 (+ 871us) server_negotiation.cc:407] Connection header received
0928 14:36:29.172487 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:29.172491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:29.172552 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:29.172649 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:29.173789 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.174321 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.175496 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.175732 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.175843 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.179564 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:49559: BlockingRecv error: recv got EOF from 127.6.23.194:49559 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:36:29.180434 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:29.170510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:49559)
0928 14:36:29.170959 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:29.171002 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:29.172241 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:29.172827 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:29.172833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:29.173183 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:29.173643 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.173656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.174459 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.174462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.175365 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.175374 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.175888 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.175892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.176054 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.176668 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:29.176684 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:29.179431 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":119,"threads_started":1}
W20260928 14:36:29.180913 15279 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:39511 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.23.254:39511: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:29.186496 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:29.181360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:47385)
0928 14:36:29.181495 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:29.181510 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:29.181604 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:29.181850 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:29.181853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:29.182010 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:29.182094 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.182099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.182894 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.182899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.183519 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.183527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.183965 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.183969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.184064 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.184520 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:29.184532 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:29.186399 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 14:36:29.186746 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:29.181433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:47385 (local address 127.6.23.254:39511)
0928 14:36:29.181553 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:29.181557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:29.181569 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:36:29.181709 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:29.181712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:29.181757 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:29.181835 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:29.182203 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.182793 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.183632 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.183853 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.183963 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.186592 (+ 2629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:47385: BlockingRecv error: recv got EOF from 127.6.23.194:47385 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:29.192396 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:29.187456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:50245)
0928 14:36:29.187562 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:29.187578 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:29.187654 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:29.187888 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:29.187891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:29.188043 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:29.188106 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.188111 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.188814 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.188820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.189342 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.189349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.189805 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.189809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.189887 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.190365 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:29.190377 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:29.192213 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 14:36:29.192427 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:29.187529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:50245 (local address 127.6.23.254:39511)
0928 14:36:29.187638 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:29.187641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:29.187653 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:36:29.187746 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:29.187749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:29.187786 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:29.187875 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:29.188214 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.188711 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.189467 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.189698 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.189805 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.192327 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:50245: BlockingRecv error: recv got EOF from 127.6.23.194:50245 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 14:36:29.192832 15279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:36:29.293027 15282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:29.293365 15282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:29.293442 15282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:29.297524 15282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:29.297632 15282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:29.297729 15282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:29.303035 15282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/krb5kdc/oryx_127.6.23.195.keytab
--principal=oryx/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:40677
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=43681
--tserver_master_addrs=127.6.23.254:39511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:29.304179 15282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:29.305235 15282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:29.314707 15290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:29.314770 15287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:29.314770 15288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:29.314908 15282 server_base.cc:1034] running on GCE node
I20260928 14:36:29.315508 15282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:29.316280 15282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:29.317500 15282 hybrid_clock.cc:648] HybridClock initialized: now 1790606189317483 us; error 27 us; skew 500 ppm
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606189, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:29.320514 15282 init.cc:377] Logged in from keytab as oryx/127.6.23.195@KRBTEST.COM (short username oryx)
I20260928 14:36:29.321370 15282 webserver.cc:460] Webserver started at http://127.6.23.195:43681/ using document root <none> and password file <none>
I20260928 14:36:29.321964 15282 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:29.322067 15282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:29.326073 15282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:29.328482 15297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:29.329770 15282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:36:29.329888 15282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "5c7c8dc80da24357a2a4ba057b6440fe"
format_stamp: "Formatted at 2026-09-28 14:36:28 on dist-test-slave-06lh"
server_key: "a71e1804beb8ac274772b19c2ce5de22"
server_key_iv: "f4a852c6808f1fee8a2f97f203b94c84"
server_key_version: "encryptionkey@0"
I20260928 14:36:29.330458 15282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:29.345228 15282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:29.348630 15282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:29.348790 15282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:29.349344 15282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:29.350157 15282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:29.350206 15282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:29.350248 15282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:29.350263 15282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:29.359429 15282 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:40677
I20260928 14:36:29.359503 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:40677 every 8 connection(s)
I20260928 14:36:29.360499 15282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:29.365557 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 15282
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:36:29.372665 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:29.362355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:47347 (local address 127.6.23.254:39511)
0928 14:36:29.362530 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:29.362535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:29.363453 (+ 918us) server_negotiation.cc:407] Connection header received
0928 14:36:29.364741 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:29.364744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:29.364792 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:29.364873 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:29.366494 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.367048 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.368246 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.368587 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.368704 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.372558 (+ 3854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:47347: BlockingRecv error: recv got EOF from 127.6.23.195:47347 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:36:29.373495 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:29.362787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:47347)
0928 14:36:29.363288 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:29.363338 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:29.364538 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:29.365063 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:29.365070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:29.365604 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:29.366309 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.366331 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.367179 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.367181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.368135 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.368143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.368749 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.368753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.368935 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.369560 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:29.369576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:29.372445 (+ 2869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":121,"threads_started":1}
W20260928 14:36:29.373953 15411 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:39511 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.23.254:39511: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
I20260928 14:36:29.375597 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:29.366630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.6.23.254:39511)
0928 14:36:29.366856 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:29.366859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:29.366869 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:36:29.366904 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:29.366906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:29.366939 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:29.367035 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:29.367846 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.368478 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.369316 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.369617 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.369695 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.371111 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:29.371117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:29.371132 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:29.371158 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:29.373245 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:29.373764 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:29.373772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:29.373780 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:29.373930 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:29.374229 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:29.374232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:29.374234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:29.374739 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:29.374921 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:29.375129 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 14:36:29.375427 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:29.380172 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:29.374421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:34293)
0928 14:36:29.374546 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:29.374561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:29.374634 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:29.375347 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:29.375353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:29.375511 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:29.375626 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.375636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.376615 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.376618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.377215 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.377224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.377714 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.377717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.377835 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.378345 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:29.378366 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:29.380079 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 14:36:29.380331 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:29.374495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:34293 (local address 127.6.23.254:39511)
0928 14:36:29.374603 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:29.374607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:29.374620 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:36:29.374701 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:29.374704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:29.374742 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:29.374823 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:29.375762 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.376477 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.377334 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.377590 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.377714 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.380209 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:34293: BlockingRecv error: recv got EOF from 127.6.23.195:34293 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:29 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:29.385597 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:29.380805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:50339)
0928 14:36:29.380928 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:29.380962 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:29.381047 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:29.381273 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:29.381276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:29.381447 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:29.381518 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.381522 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.382230 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.382235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.382758 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:29.382766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.383214 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.383217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:29.383329 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.383776 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:29.383787 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:29.385504 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 14:36:29.385735 15148 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:29.380894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:50339 (local address 127.6.23.254:39511)
0928 14:36:29.381052 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:29.381056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:29.381071 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:29.381135 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:29.381137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:29.381178 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:29.381260 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:29.381629 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.382126 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.382876 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:29.383106 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:29.383213 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:29.385625 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:50339: BlockingRecv error: recv got EOF from 127.6.23.195:50339 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 14:36:29.385953 15411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 14:36:30 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:30 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:30.011787 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:30.005451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:36119)
0928 14:36:30.005740 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:30.005759 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:30.005878 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:30.006456 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:30.006459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:30.006712 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:30.006851 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:30.006858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.007850 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.007854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:30.008433 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:30.008441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.008905 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.008909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:30.009008 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:30.009519 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:30.009530 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:30.011657 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
W20260928 14:36:30.011970 15417 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:30.005609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:36119 (local address 127.6.23.254:39511)
0928 14:36:30.005988 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:30.005993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:30.006011 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:36:30.006060 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:30.006063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:30.006115 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:30.006223 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:30.006977 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.007745 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.008560 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.008791 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.008903 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:30.011820 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:36119: BlockingRecv error: recv got EOF from 127.6.23.193:36119 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
Sep 28 14:36:30 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:30 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:30.199542 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:30.193618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:41989)
0928 14:36:30.193919 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:30.193934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:30.194038 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:30.194301 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:30.194304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:30.194492 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:30.194588 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:30.194594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.195476 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.195483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:30.196374 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:30.196389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.196839 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.196842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:30.196928 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:30.197491 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:30.197512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:30.199418 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":69,"threads_started":1}
W20260928 14:36:30.199683 15417 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:30.193702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:41989 (local address 127.6.23.254:39511)
0928 14:36:30.193903 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:30.193906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:30.194010 (+ 104us) server_negotiation.cc:407] Connection header received
0928 14:36:30.194122 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:30.194124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:30.194167 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:30.194236 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:30.194709 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.195323 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.196488 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.196716 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.196830 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:30.199568 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:41989: BlockingRecv error: recv got EOF from 127.6.23.194:41989 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 14:36:30 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:30 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:30.391868 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:30.386528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:43739)
0928 14:36:30.386786 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:30.386799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:30.386895 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:30.387136 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:30.387138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:30.387343 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:30.387467 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:30.387473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.388249 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.388252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:30.388840 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:30.388849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.389316 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.389320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:30.389403 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:30.389866 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:30.389877 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:30.391751 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":67,"threads_started":1}
W20260928 14:36:30.392062 15417 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:30.386653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:43739 (local address 127.6.23.254:39511)
0928 14:36:30.386811 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:30.386816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:30.386849 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:36:30.386987 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:30.386989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:30.387033 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:30.387099 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:30.387599 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.388141 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.388955 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:30.389174 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:30.389266 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:30.391885 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:43739: BlockingRecv error: recv got EOF from 127.6.23.195:43739 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 14:36:31 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:31 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:31.018205 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:31.012611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:32965)
0928 14:36:31.012948 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:31.012962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:31.013057 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:31.013309 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:31.013311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:31.013490 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:31.013585 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:31.013590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.014329 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.014332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:31.014899 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:31.014907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.015373 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.015376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:31.015446 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:31.015937 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:31.015949 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:31.018007 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
W20260928 14:36:31.018265 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:31.012753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:32965 (local address 127.6.23.254:39511)
0928 14:36:31.012999 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:31.013003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:31.013016 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:36:31.013151 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:31.013153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:31.013195 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:31.013273 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:31.013683 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.014222 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.015009 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.015244 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.015365 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:31.018137 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:32965: BlockingRecv error: recv got EOF from 127.6.23.193:32965 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":58,"threads_started":1}
Sep 28 14:36:31 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:31 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:31.206648 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:31.200652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:50091)
0928 14:36:31.200936 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:31.200952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:31.201081 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:31.201559 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:31.201565 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:31.201805 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:31.201955 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:31.201963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.202787 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.202790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:31.203397 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:31.203410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.203858 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.203861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:31.203930 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:31.204430 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:31.204442 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:31.206502 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":105,"threads_started":1}
W20260928 14:36:31.206820 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:31.200754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:50091 (local address 127.6.23.254:39511)
0928 14:36:31.200914 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:31.200918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:31.201089 (+ 171us) server_negotiation.cc:407] Connection header received
0928 14:36:31.201216 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:31.201218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:31.201269 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:31.201508 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:31.202093 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.202674 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.203525 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.203753 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.203838 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:31.206682 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:50091: BlockingRecv error: recv got EOF from 127.6.23.194:50091 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 14:36:31 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:31 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:31.398371 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:31.392735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:43113)
0928 14:36:31.393027 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:31.393046 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:31.393167 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:31.393468 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:31.393471 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:31.393673 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:31.393769 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:31.393774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.394550 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.394555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:31.395113 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:31.395121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.395626 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.395630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:31.395716 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:31.396223 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:31.396235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:31.398195 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
W20260928 14:36:31.398474 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:31.392962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:43113 (local address 127.6.23.254:39511)
0928 14:36:31.393115 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:31.393119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:31.393134 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:31.393286 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:31.393290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:31.393346 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:31.393435 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:31.393903 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.394439 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.395278 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:31.395506 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:31.395623 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:31.398331 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:43113: BlockingRecv error: recv got EOF from 127.6.23.195:43113 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 14:36:32 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:32 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:32.025017 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:32.019097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:42165)
0928 14:36:32.019469 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:32.019483 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:32.019612 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:32.020013 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:32.020017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:32.020293 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:32.020399 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:32.020405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.021222 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.021225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:32.021798 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:32.021806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.022300 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.022303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:32.022394 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:32.022918 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:32.022930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:32.024887 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":88,"threads_started":1}
W20260928 14:36:32.025240 15425 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:32.019254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:42165 (local address 127.6.23.254:39511)
0928 14:36:32.019497 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:32.019501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:32.019540 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:36:32.019775 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:32.019778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:32.019835 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:32.019956 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:32.020548 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.021108 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.021929 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.022172 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.022272 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:32.025087 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:42165: BlockingRecv error: recv got EOF from 127.6.23.193:42165 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
Sep 28 14:36:32 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:32 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:32.213654 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:32.207914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:37403)
0928 14:36:32.208225 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:32.208238 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:32.208372 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:32.208676 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:32.208679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:32.208893 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:32.209001 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:32.209007 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.209785 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.209792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:32.210370 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:32.210377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.210849 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.210854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:32.210951 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:32.211592 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:32.211604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:32.213536 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
W20260928 14:36:32.213826 15425 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:32.207940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:37403 (local address 127.6.23.254:39511)
0928 14:36:32.208124 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:32.208129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:32.208317 (+ 188us) server_negotiation.cc:407] Connection header received
0928 14:36:32.208501 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:32.208504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:32.208560 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:32.208652 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:32.209121 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.209671 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.210505 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.210735 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.210847 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:32.213695 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:37403: BlockingRecv error: recv got EOF from 127.6.23.194:37403 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 14:36:32 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:32 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:32.404810 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:32.399218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:43795)
0928 14:36:32.399530 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:32.399543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:32.399667 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:32.399968 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:32.399971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:32.400176 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:32.400269 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:32.400275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.401061 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.401066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:32.401648 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:32.401656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.402122 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.402125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:32.402212 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:32.402776 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:32.402788 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:32.404630 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
W20260928 14:36:32.404872 15425 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:32.399303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:43795 (local address 127.6.23.254:39511)
0928 14:36:32.399473 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:32.399478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:32.399617 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:36:32.399784 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:32.399789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:32.399848 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:32.399934 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:32.400391 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.400947 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.401773 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:32.402010 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:32.402119 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:32.404757 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:43795: BlockingRecv error: recv got EOF from 127.6.23.195:43795 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 14:36:33 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:33 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:33.031363 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:33.025844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:52573)
0928 14:36:33.026133 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:33.026147 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:33.026246 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:33.026502 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:33.026505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:33.026683 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:33.026781 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:33.026786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.027626 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.027633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:33.028219 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:33.028228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.028649 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.028652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:33.028714 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:33.029265 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:33.029277 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:33.031150 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
W20260928 14:36:33.031483 15429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:33.025902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:52573 (local address 127.6.23.254:39511)
0928 14:36:33.026164 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:33.026167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:33.026197 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:36:33.026339 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:33.026341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:33.026384 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:33.026463 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:33.026890 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.027521 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.028321 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.028552 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.028627 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:33.031320 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:52573: BlockingRecv error: recv got EOF from 127.6.23.193:52573 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
Sep 28 14:36:33 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:33 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:33.228401 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:33.214523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:44705)
0928 14:36:33.214923 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:33.214937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:33.215036 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:33.215290 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:33.215292 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:33.215480 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:33.215581 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:33.215586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.216320 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.216323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:33.217029 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:33.217038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.217561 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.217564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:33.217625 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:33.226155 (+ 8530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:33.226171 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:33.228207 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":210,"threads_started":1}
W20260928 14:36:33.228521 15429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:33.214647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:44705 (local address 127.6.23.254:39511)
0928 14:36:33.214807 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:33.214810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:33.215007 (+ 197us) server_negotiation.cc:407] Connection header received
0928 14:36:33.215117 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:33.215119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:33.215163 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:33.215278 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:33.215681 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.216214 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.217150 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.217449 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.217536 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:33.228373 (+ 10837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:44705: BlockingRecv error: recv got EOF from 127.6.23.194:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 14:36:33 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:33 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:33.410983 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:33.405572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:48329)
0928 14:36:33.405838 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:33.405852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:33.405946 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:33.406184 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:33.406186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:33.406386 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:33.406487 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:33.406493 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.407210 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.407213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:33.407851 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:33.407860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.408313 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.408315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:33.408406 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:33.408937 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:33.408949 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:33.410861 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
W20260928 14:36:33.411291 15429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:33.405721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:48329 (local address 127.6.23.254:39511)
0928 14:36:33.405880 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:33.405884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:33.405902 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:36:33.406032 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:33.406034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:33.406078 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:33.406143 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:33.406589 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.407109 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.407980 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:33.408202 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:33.408290 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:33.411068 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:48329: BlockingRecv error: recv got EOF from 127.6.23.195:48329 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 14:36:34 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:34 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:34.037693 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:34.032194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:32891)
0928 14:36:34.032501 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:34.032515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:34.032602 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:34.032892 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:34.032894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:34.033081 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:34.033175 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:34.033180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.034072 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.034075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:34.034665 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:34.034674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.035178 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.035181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:34.035264 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:34.035724 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:34.035736 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:34.037568 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
W20260928 14:36:34.037925 15433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:34.032295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:32891 (local address 127.6.23.254:39511)
0928 14:36:34.032570 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:34.032574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:34.032588 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:36:34.032724 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:34.032726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:34.032775 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:34.032858 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:34.033305 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.033965 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.034780 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.035075 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.035158 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:34.037771 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:32891: BlockingRecv error: recv got EOF from 127.6.23.193:32891 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
Sep 28 14:36:34 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:34 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:34.235462 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:34.229459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:52899)
0928 14:36:34.229794 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:34.229810 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:34.229921 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:34.230200 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:34.230202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:34.230477 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:34.230586 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:34.230593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.231444 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.231447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:34.232038 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:34.232046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.232521 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.232525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:34.232613 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:34.233207 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:34.233219 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:34.235334 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":83,"threads_started":1}
W20260928 14:36:34.235764 15433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:34.229587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:52899 (local address 127.6.23.254:39511)
0928 14:36:34.229708 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:34.229713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:34.229863 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:36:34.230031 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:34.230035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:34.230090 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:34.230176 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:34.230708 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.231319 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.232167 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.232409 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.232519 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:34.235501 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:52899: BlockingRecv error: recv got EOF from 127.6.23.194:52899 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 14:36:34 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:34 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:34.417407 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:34.411859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:49065)
0928 14:36:34.412139 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:34.412152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:34.412243 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:34.412529 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:34.412532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:34.412711 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:34.412818 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:34.412823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.413636 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.413642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:34.414233 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:34.414241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.414692 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.414695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:34.414761 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:34.415334 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:34.415346 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:34.417230 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":109,"threads_started":1}
W20260928 14:36:34.417503 15433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:34.411955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:49065 (local address 127.6.23.254:39511)
0928 14:36:34.412092 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:34.412096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:34.412219 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:36:34.412311 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:34.412313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:34.412388 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:34.412459 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:34.412955 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.413503 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.414341 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:34.414577 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:34.414655 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:34.417353 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:49065: BlockingRecv error: recv got EOF from 127.6.23.195:49065 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 14:36:35 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:35 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:35.044133 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:35.038518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:33105)
0928 14:36:35.038839 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:35.038894 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:35.039014 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:35.039306 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:35.039312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:35.039517 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:35.039618 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:35.039623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.040436 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.040439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:35.040997 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:35.041005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.041422 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.041424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:35.041487 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:35.042078 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:35.042091 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:35.043954 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":131,"threads_started":1}
W20260928 14:36:35.044217 15437 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:35.038612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:33105 (local address 127.6.23.254:39511)
0928 14:36:35.038846 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:35.038850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:35.038974 (+ 124us) server_negotiation.cc:407] Connection header received
0928 14:36:35.039095 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:35.039097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:35.039141 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:35.039283 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:35.039722 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.040332 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.041103 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.041322 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.041399 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:35.044089 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:33105: BlockingRecv error: recv got EOF from 127.6.23.193:33105 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":86,"threads_started":1}
Sep 28 14:36:35 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:35 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:35.242347 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:35.236434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:40815)
0928 14:36:35.236785 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:35.236800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:35.236924 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:35.237200 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:35.237203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:35.237466 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:35.237574 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:35.237581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.238357 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.238371 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:35.238955 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:35.238963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.239482 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.239486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:35.239573 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:35.240269 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:35.240284 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:35.242188 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":105,"threads_started":1}
W20260928 14:36:35.242558 15437 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:35.236456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:40815 (local address 127.6.23.254:39511)
0928 14:36:35.236604 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:35.236607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:35.236874 (+ 267us) server_negotiation.cc:407] Connection header received
0928 14:36:35.237022 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:35.237024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:35.237074 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:35.237174 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:35.237699 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.238241 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.239079 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.239369 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.239480 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:35.242375 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:40815: BlockingRecv error: recv got EOF from 127.6.23.194:40815 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:36:35 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:35 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:35.423527 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:35.418202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:33343)
0928 14:36:35.418520 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:35.418533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:35.418626 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:35.418855 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:35.418857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:35.419033 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:35.419122 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:35.419127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.419929 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.419932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:35.420520 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:35.420529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.420964 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.420967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:35.421032 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:35.421523 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:35.421536 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:35.423399 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
W20260928 14:36:35.423717 15437 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:35.418238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:33343 (local address 127.6.23.254:39511)
0928 14:36:35.418404 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:35.418411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:35.418600 (+ 189us) server_negotiation.cc:407] Connection header received
0928 14:36:35.418704 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:35.418706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:35.418752 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:35.418819 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:35.419213 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.419804 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.420635 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:35.420859 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:35.420936 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:35.423587 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:33343: BlockingRecv error: recv got EOF from 127.6.23.195:33343 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 14:36:36 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:36 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:36.050613 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:36.045005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:38863)
0928 14:36:36.045337 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:36.045351 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:36.045453 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:36.045700 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:36.045703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:36.045886 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:36.046012 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:36.046017 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.046940 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.046944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:36.047545 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:36.047553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.047992 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.047995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:36.048055 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:36.048652 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:36.048664 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:36.050493 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
W20260928 14:36:36.050827 15441 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:36.045117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:38863 (local address 127.6.23.254:39511)
0928 14:36:36.045362 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:36.045365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:36.045401 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:36:36.045536 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:36.045538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:36.045582 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:36.045670 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:36.046116 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.046830 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.047646 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.047865 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.047970 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:36.050651 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:38863: BlockingRecv error: recv got EOF from 127.6.23.193:38863 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
Sep 28 14:36:36 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:36 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:36.248772 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:36.243072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:42991)
0928 14:36:36.243438 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:36.243451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:36.243590 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:36.243907 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:36.243910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:36.244129 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:36.244221 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:36.244227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.245072 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.245077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:36.245648 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:36.245655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.246128 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.246132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:36.246237 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:36.246725 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:36.246737 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:36.248653 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":104,"threads_started":1}
W20260928 14:36:36.248961 15441 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:36.243191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:42991 (local address 127.6.23.254:39511)
0928 14:36:36.243380 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:36.243414 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:36.243555 (+ 141us) server_negotiation.cc:407] Connection header received
0928 14:36:36.243706 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:36.243710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:36.243770 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:36.243874 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:36.244359 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.244958 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.245770 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.246013 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.246122 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:36.248801 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:42991: BlockingRecv error: recv got EOF from 127.6.23.194:42991 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 14:36:36 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:36 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:36.430156 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:36.424498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:48227)
0928 14:36:36.424856 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:36.424871 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:36.424982 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:36.425203 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:36.425204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:36.425378 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:36.425471 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:36.425476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.426190 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.426192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:36.426732 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:36.426738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.427159 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.427162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:36.427223 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:36.427808 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:36.427819 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:36.429986 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":101,"threads_started":1}
W20260928 14:36:36.430214 15441 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:36.424540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:48227 (local address 127.6.23.254:39511)
0928 14:36:36.424702 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:36.424707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:36.424958 (+ 251us) server_negotiation.cc:407] Connection header received
0928 14:36:36.425052 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:36.425054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:36.425101 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:36.425167 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:36.425575 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.426100 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.426831 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:36.427056 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:36.427133 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:36.430107 (+ 2974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:48227: BlockingRecv error: recv got EOF from 127.6.23.195:48227 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 14:36:37 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:37 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:37.057089 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:37.051492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:54039)
0928 14:36:37.051823 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:37.051836 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:37.051961 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:37.052274 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:37.052277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:37.052485 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:37.052581 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:37.052587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.053337 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.053340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:37.053935 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:37.053952 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.054499 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.054501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:37.054565 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:37.055135 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:37.055146 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:37.056973 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
W20260928 14:36:37.057260 15445 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:37.051646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:54039 (local address 127.6.23.254:39511)
0928 14:36:37.051880 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:37.051886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:37.051905 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:36:37.052080 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:37.052086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:37.052145 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:37.052238 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:37.052679 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.053221 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.054069 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.054397 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.054480 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:37.057118 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:54039: BlockingRecv error: recv got EOF from 127.6.23.193:54039 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
Sep 28 14:36:37 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:37 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:37.255861 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:37.249947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:35387)
0928 14:36:37.250225 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:37.250241 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:37.250432 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:37.250748 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:37.250751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:37.250997 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:37.251128 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:37.251140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.252090 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.252093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:37.252685 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:37.252693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.253109 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.253112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:37.253237 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:37.253837 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:37.253849 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:37.255741 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":83,"threads_started":1}
W20260928 14:36:37.256007 15445 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:37.250021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:35387 (local address 127.6.23.254:39511)
0928 14:36:37.250169 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:37.250173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:37.250330 (+ 157us) server_negotiation.cc:407] Connection header received
0928 14:36:37.250556 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:37.250560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:37.250617 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:37.250705 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:37.251283 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.251980 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.252792 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.253011 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.253086 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:37.255884 (+ 2798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:35387: BlockingRecv error: recv got EOF from 127.6.23.194:35387 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 14:36:37 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:37 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:36:37.437031 15445 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:37.431151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:44975 (local address 127.6.23.254:39511)
0928 14:36:37.431355 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:37.431359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:37.431499 (+ 140us) server_negotiation.cc:407] Connection header received
0928 14:36:37.431665 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:37.431668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:37.431724 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:37.431814 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:37.432265 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.432807 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.433625 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.433859 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.433980 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:37.436864 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:44975: BlockingRecv error: recv got EOF from 127.6.23.195:44975 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 14:36:37.437105 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:37.431016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:44975)
0928 14:36:37.431404 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:37.431420 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:37.431563 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:37.431833 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:37.431836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:37.432051 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:37.432144 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:37.432150 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.432939 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.432943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:37.433504 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:37.433511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:37.433983 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:37.433987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:37.434074 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:37.434676 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:37.434688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:37.436660 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":86,"threads_started":1}
Sep 28 14:36:38 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:38 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:38.063612 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:38.057883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:57853)
0928 14:36:38.058291 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:38.058305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:38.058434 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:38.058691 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:38.058693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:38.058880 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:38.058981 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:38.058986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.059743 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.059746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:38.060299 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:38.060307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.060724 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.060727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:38.060785 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:38.061352 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:38.061364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:38.063468 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":111,"threads_started":1}
W20260928 14:36:38.063819 15449 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:38.057939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:57853 (local address 127.6.23.254:39511)
0928 14:36:38.058307 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:38.058310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:38.058363 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:36:38.058526 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:38.058528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:38.058573 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:38.058651 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:38.059082 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.059644 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.060403 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.060625 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.060703 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:38.063657 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:57853: BlockingRecv error: recv got EOF from 127.6.23.193:57853 (error 108)
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":191,"threads_started":1}
Sep 28 14:36:38 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:38 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:38.268213 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:38.256808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:42809)
0928 14:36:38.257076 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:38.257089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:38.257225 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:38.257472 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:38.257492 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:38.257670 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:38.257774 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:38.257780 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.258518 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.258520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:38.259074 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:38.259082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.259613 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.259615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:38.259681 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:38.265895 (+ 6214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:38.265909 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:38.267993 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
W20260928 14:36:38.268455 15449 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:38.256866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:42809 (local address 127.6.23.254:39511)
0928 14:36:38.257017 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:38.257022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:38.257166 (+ 144us) server_negotiation.cc:407] Connection header received
0928 14:36:38.257321 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:38.257323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:38.257368 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:38.257434 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:38.257868 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.258403 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.259237 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.259515 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.259595 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:38.268224 (+ 8629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:42809: BlockingRecv error: recv got EOF from 127.6.23.194:42809 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 14:36:38 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:38 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:38.443313 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:38.437861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:34369)
0928 14:36:38.438125 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:38.438137 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:38.438235 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:38.438510 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:38.438513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:38.438751 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:38.438872 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:38.438878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.439651 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.439653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:38.440299 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:38.440308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.440763 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.440765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:38.440832 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:38.441346 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:38.441358 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:38.443169 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
W20260928 14:36:38.443573 15449 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:38.437977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:34369 (local address 127.6.23.254:39511)
0928 14:36:38.438112 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:38.438115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:38.438200 (+ 85us) server_negotiation.cc:407] Connection header received
0928 14:36:38.438300 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:38.438302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:38.438346 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:38.438442 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:38.438976 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.439546 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.440428 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:38.440654 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:38.440735 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:38.443455 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:34369: BlockingRecv error: recv got EOF from 127.6.23.195:34369 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 14:36:39 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:39 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:39.069948 15452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:39.064554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:55783)
0928 14:36:39.064845 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:39.064858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:39.064942 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:39.065209 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:39.065212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:39.065408 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:39.065507 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:39.065512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.066301 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.066304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:39.066904 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:39.066914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.067356 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.067358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:39.067451 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:39.067945 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:39.067957 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:39.069824 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":112,"threads_started":1}
W20260928 14:36:39.070109 15453 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:39.064623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:55783 (local address 127.6.23.254:39511)
0928 14:36:39.064893 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:39.064896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:39.064925 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:36:39.065034 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:39.065036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:39.065077 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:39.065159 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:39.065630 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.066180 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.067015 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.067249 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.067337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:39.069973 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:55783: BlockingRecv error: recv got EOF from 127.6.23.193:55783 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
Sep 28 14:36:39 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:39 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:39.275048 15454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:39.269126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:47745)
0928 14:36:39.269468 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:39.269483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:39.269583 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:39.269809 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:39.269811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:39.270005 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:39.270105 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:39.270111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.270895 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.270898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:39.271557 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:39.271568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.272132 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.272135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:39.272229 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:39.272903 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:39.272916 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:39.274929 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":85,"threads_started":1}
W20260928 14:36:39.275197 15453 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:39.269240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:47745 (local address 127.6.23.254:39511)
0928 14:36:39.269476 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:39.269480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:39.269544 (+ 64us) server_negotiation.cc:407] Connection header received
0928 14:36:39.269655 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:39.269657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:39.269700 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:39.269770 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:39.270220 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.270777 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.271683 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.271986 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.272092 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:39.275075 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:47745: BlockingRecv error: recv got EOF from 127.6.23.194:47745 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 14:36:39 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:39 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:39.450562 15455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:39.444263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:55821)
0928 14:36:39.444548 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:39.444561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:39.444644 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:39.444926 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:39.444929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:39.445109 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:39.445208 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:39.445213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.446060 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.446064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:39.446726 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:39.446736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.447284 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.447289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:39.447417 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:39.448002 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:39.448015 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:39.450373 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":111,"threads_started":1}
W20260928 14:36:39.450685 15453 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:39.444453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:55821 (local address 127.6.23.254:39511)
0928 14:36:39.444593 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:39.444597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:39.444610 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:36:39.444723 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:39.444725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:39.444770 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:39.444843 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:39.445312 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.445887 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.446848 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:39.447091 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:39.447179 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:39.450558 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:55821: BlockingRecv error: recv got EOF from 127.6.23.195:55821 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 14:36:40 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:40 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:40.076181 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:40.070863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:36227)
0928 14:36:40.071168 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:40.071182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:40.071348 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:40.071608 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:40.071611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:40.071791 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:40.071890 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:40.071895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.072644 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.072647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:40.073233 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:40.073241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.073656 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.073658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:40.073719 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:40.074201 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:40.074212 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:40.076061 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":102,"threads_started":1}
W20260928 14:36:40.076347 15457 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:40.070996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:36227 (local address 127.6.23.254:39511)
0928 14:36:40.071246 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:40.071250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:40.071262 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:36:40.071443 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:40.071445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:40.071491 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:40.071565 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:40.072001 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.072540 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.073336 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.073560 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.073633 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:40.076209 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:36227: BlockingRecv error: recv got EOF from 127.6.23.193:36227 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":64,"threads_started":1}
Sep 28 14:36:40 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:40 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:40.281612 15458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:40.275866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:33319)
0928 14:36:40.276204 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:40.276221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:40.276379 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:40.276660 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:40.276663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:40.276882 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:40.276983 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:40.276989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.277751 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.277756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:40.278348 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:40.278356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.278817 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.278821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:40.278919 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:40.279460 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:40.279472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:40.281401 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":88,"threads_started":1}
W20260928 14:36:40.281670 15457 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:40.276011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:33319 (local address 127.6.23.254:39511)
0928 14:36:40.276197 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:40.276202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:40.276312 (+ 110us) server_negotiation.cc:407] Connection header received
0928 14:36:40.276490 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:40.276494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:40.276548 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:40.276639 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:40.277102 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.277640 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.278470 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.278697 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.278802 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:40.281558 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:33319: BlockingRecv error: recv got EOF from 127.6.23.194:33319 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 14:36:40 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:40 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:40.457720 15459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:40.451437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:59701)
0928 14:36:40.451886 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:40.451900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:40.452064 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:40.452456 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:40.452459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:40.452675 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:40.452818 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:40.452838 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.453745 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.453748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:40.454476 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:40.454485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.454959 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.454962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:40.455026 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:40.455570 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:40.455582 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:40.457589 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":123,"threads_started":1}
W20260928 14:36:40.457926 15457 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:40.451619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:59701 (local address 127.6.23.254:39511)
0928 14:36:40.451808 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:40.451813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:40.452004 (+ 191us) server_negotiation.cc:407] Connection header received
0928 14:36:40.452194 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:40.452197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:40.452275 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:40.452358 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:40.452952 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.453632 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.454598 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:40.454849 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:40.454934 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:40.457767 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:59701: BlockingRecv error: recv got EOF from 127.6.23.195:59701 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 14:36:41 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:41 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:41.082973 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:41.077285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:55415)
0928 14:36:41.077659 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:41.077675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:41.077790 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:41.078101 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:41.078106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:41.078301 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:41.078402 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:41.078407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.079207 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.079210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:41.079802 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:41.079810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.080344 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.080348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:41.080421 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:41.081017 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:41.081029 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:41.082840 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":140,"threads_started":1}
W20260928 14:36:41.083214 15461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:41.077395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:55415 (local address 127.6.23.254:39511)
0928 14:36:41.077673 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:41.077680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:41.077767 (+ 87us) server_negotiation.cc:407] Connection header received
0928 14:36:41.077885 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:41.077887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:41.077949 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:41.078049 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:41.078523 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.079108 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.079951 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.080205 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.080304 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:41.083054 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:55415: BlockingRecv error: recv got EOF from 127.6.23.193:55415 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":64,"threads_started":1}
Sep 28 14:36:41 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:41 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:41.288264 15462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:41.282510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:58455)
0928 14:36:41.282790 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:41.282804 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:41.282902 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:41.283136 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:41.283138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:41.283342 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:41.283458 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:41.283463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.284252 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.284255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:41.284847 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:41.284855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.285339 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.285342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:41.285435 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:41.285974 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:41.285989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:41.288141 (+ 2152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
W20260928 14:36:41.288472 15461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:41.282550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:58455 (local address 127.6.23.254:39511)
0928 14:36:41.282690 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:41.282695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:41.282903 (+ 208us) server_negotiation.cc:407] Connection header received
0928 14:36:41.282981 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:41.282983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:41.283030 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:41.283102 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:41.283574 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.284126 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.284996 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.285224 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.285313 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:41.288318 (+ 3005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:58455: BlockingRecv error: recv got EOF from 127.6.23.194:58455 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 14:36:41 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:41 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:41.464213 15463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:41.458753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:54161)
0928 14:36:41.459031 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:41.459046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:41.459157 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:41.459420 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:41.459422 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:41.459605 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:41.459696 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:41.459701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.460582 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.460589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:41.461231 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:41.461241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.461670 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.461672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:41.461731 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:41.462203 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:41.462214 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:41.464097 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
W20260928 14:36:41.464461 15461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:41.458791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:54161 (local address 127.6.23.254:39511)
0928 14:36:41.458948 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:41.458953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:41.459129 (+ 176us) server_negotiation.cc:407] Connection header received
0928 14:36:41.459258 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:41.459260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:41.459309 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:41.459381 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:41.459842 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.460490 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.461330 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:41.461577 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:41.461657 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:41.464287 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:54161: BlockingRecv error: recv got EOF from 127.6.23.195:54161 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 14:36:42 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:42 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:42.089527 15464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:42.083835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:52629)
0928 14:36:42.084127 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:42.084140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:42.084266 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:42.084582 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:42.084586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:42.084832 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:42.084931 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:42.084938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.085723 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.085727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:42.086314 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:42.086351 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.086833 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.086836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:42.086908 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:42.087572 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:42.087585 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:42.089388 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
W20260928 14:36:42.089701 15465 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:42.083924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:52629 (local address 127.6.23.254:39511)
0928 14:36:42.084197 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:42.084202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:42.084223 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:36:42.084378 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:42.084381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:42.084436 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:42.084534 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:42.085047 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.085605 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.086476 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.086720 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.086808 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:42.089560 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:52629: BlockingRecv error: recv got EOF from 127.6.23.193:52629 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
Sep 28 14:36:42 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:42 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:36:42.295027 15465 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:42.289276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:57957 (local address 127.6.23.254:39511)
0928 14:36:42.289434 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:42.289437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:42.289517 (+ 80us) server_negotiation.cc:407] Connection header received
0928 14:36:42.289691 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:42.289694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:42.289743 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:42.289821 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:42.290339 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.290907 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.291777 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.291993 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.292073 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:42.294929 (+ 2856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:57957: BlockingRecv error: recv got EOF from 127.6.23.194:57957 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:36:42.294921 15466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:42.289119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:57957)
0928 14:36:42.289432 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:42.289447 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:42.289546 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:42.289892 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:42.289895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:42.290087 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:42.290195 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:42.290201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.291016 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.291019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:42.291677 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:42.291686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.292087 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.292090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:42.292152 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:42.292738 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:42.292749 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:42.294763 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
Sep 28 14:36:42 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:42 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:42.470485 15467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:42.465021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:35473)
0928 14:36:42.465277 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:42.465305 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:42.465416 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:42.465695 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:42.465697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:42.465936 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:42.466045 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:42.466051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.466833 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.466837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:42.467462 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:42.467471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.467931 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.467935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:42.468019 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:42.468536 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:42.468548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:42.470360 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
W20260928 14:36:42.470670 15465 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:42.465199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:35473 (local address 127.6.23.254:39511)
0928 14:36:42.465347 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:42.465353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:42.465371 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:36:42.465539 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:42.465542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:42.465588 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:42.465668 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:42.466167 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.466712 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.467585 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:42.467819 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:42.467925 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:42.470509 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:35473: BlockingRecv error: recv got EOF from 127.6.23.195:35473 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 14:36:43 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:43 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:43.096385 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:43.090375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:43295)
0928 14:36:43.090825 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:43.090841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:43.090962 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:43.091283 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:43.091286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:43.091547 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:43.091656 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:43.091662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.092479 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.092481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:43.093075 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:43.093083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.093582 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.093611 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:43.093682 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:43.094232 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:43.094243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:43.096243 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":135,"threads_started":1}
W20260928 14:36:43.096578 15469 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:43.090494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:43295 (local address 127.6.23.254:39511)
0928 14:36:43.090816 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:43.090819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:43.090913 (+ 94us) server_negotiation.cc:407] Connection header received
0928 14:36:43.091055 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:43.091058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:43.091113 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:43.091211 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:43.091784 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.092367 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.093183 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.093443 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.093523 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:43.096406 (+ 2883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:43295: BlockingRecv error: recv got EOF from 127.6.23.193:43295 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":73,"threads_started":1}
Sep 28 14:36:43 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:43 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:43.306775 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:43.296018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:39585)
0928 14:36:43.296298 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:43.296311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:43.296410 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:43.296676 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:43.296678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:43.296856 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:43.296952 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:43.296957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.297736 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.297738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:43.298299 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:43.298306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.298754 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.298757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:43.298816 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:43.304771 (+ 5955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:43.304784 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:43.306639 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":89,"threads_started":1}
W20260928 14:36:43.306988 15469 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:43.296129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:39585 (local address 127.6.23.254:39511)
0928 14:36:43.296324 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:43.296328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:43.296366 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:36:43.296522 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:43.296525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:43.296572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:43.296666 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:43.297087 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.297638 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.298421 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.298650 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.298739 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:43.306816 (+ 8077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:39585: BlockingRecv error: recv got EOF from 127.6.23.194:39585 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 14:36:43 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:43 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.195@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:43.477056 15471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:43.471422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.195:60379)
0928 14:36:43.471710 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:43.471730 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:43.471842 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:43.472140 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:43.472143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:43.472388 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:43.472497 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:43.472504 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.473324 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.473327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:43.473890 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:43.473898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.474405 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.474408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:43.474479 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:43.475048 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:43.475061 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:43.476942 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":104,"threads_started":1}
W20260928 14:36:43.477243 15469 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:43.471555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:60379 (local address 127.6.23.254:39511)
0928 14:36:43.471736 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:43.471740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:43.471810 (+ 70us) server_negotiation.cc:407] Connection header received
0928 14:36:43.471928 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:43.471931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:43.471980 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:43.472068 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:43.472622 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.473184 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.474034 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:43.474260 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:43.474345 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:43.477101 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.195:60379: BlockingRecv error: recv got EOF from 127.6.23.195:60379 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 14:36:44 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:44 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.193@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:44.102702 15472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:44.097463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.193:53583)
0928 14:36:44.097741 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:44.097755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:44.097838 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:44.098091 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:44.098093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:44.098274 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:44.098366 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:44.098371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.099091 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.099093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:44.099770 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:44.099780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.100189 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.100192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:44.100251 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:44.100798 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:44.100810 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:44.102582 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
W20260928 14:36:44.102895 15473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:44.097534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:53583 (local address 127.6.23.254:39511)
0928 14:36:44.097779 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:44.097782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:44.097800 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:36:44.097932 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:44.097935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:44.097976 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:44.098062 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:44.098464 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.098996 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.099863 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.100095 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.100175 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:44.102718 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.193:53583: BlockingRecv error: recv got EOF from 127.6.23.193:53583 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
Sep 28 14:36:44 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:36:44 dist-test-slave-06lh krb5kdc[14415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.23.194@KRBTEST.COM for oryx/127.6.23.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:36:44.313323 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:44.307778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:39511 (local address 127.6.23.194:35333)
0928 14:36:44.308076 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:44.308092 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:44.308220 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:44.308495 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:44.308498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:44.308710 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:44.308797 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:44.308802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.309560 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.309566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:44.310135 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:44.310142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.310608 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.310611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:44.310697 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:44.311263 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:44.311276 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:44.313153 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.23.254:39511: Server oryx/127.6.23.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
W20260928 14:36:44.313588 15473 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:44.307875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:35333 (local address 127.6.23.254:39511)
0928 14:36:44.308052 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:44.308057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:44.308173 (+ 116us) server_negotiation.cc:407] Connection header received
0928 14:36:44.308334 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:44.308337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:44.308389 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:44.308478 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:44.308913 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.309453 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.310250 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.310498 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.310605 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:44.313332 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.23.194:35333: BlockingRecv error: recv got EOF from 127.6.23.194:35333 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:36:44.371960 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 14998
I20260928 14:36:44.377835 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 15150
I20260928 14:36:44.383538 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 15282
I20260928 14:36:44.388912 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 14928
2026-09-28T14:36:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18053 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T14:36:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:44Z Disabled control of system clock
I20260928 14:36:44.419948 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:41241
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:41241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:36:44.540098 15478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:44.540401 15478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:44.540437 15478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:44.544458 15478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:44.544521 15478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:44.544538 15478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:44.544554 15478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:44.544569 15478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:44.549481 15478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:41241
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:41241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:44.550625 15478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:44.552058 15478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:44.561394 15484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:44.561530 15483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:44.561574 15478 server_base.cc:1034] running on GCE node
W20260928 14:36:44.561394 15486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:44.562366 15478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:44.563683 15478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:44.564870 15478 hybrid_clock.cc:648] HybridClock initialized: now 1790606204564851 us; error 35 us; skew 500 ppm
I20260928 14:36:44.566898 15478 webserver.cc:460] Webserver started at http://127.6.23.254:39689/ using document root <none> and password file <none>
I20260928 14:36:44.567569 15478 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:44.567657 15478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:44.567884 15478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:44.569988 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "1b8728b376214112bed4fb6d916713ec"
format_stamp: "Formatted at 2026-09-28 14:36:44 on dist-test-slave-06lh"
server_key: "4cabddd1d637ffec31b1e3557a20e2b3"
server_key_iv: "dabf94b79f32d066ae6916c9965f2479"
server_key_version: "encryptionkey@0"
I20260928 14:36:44.570565 15478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "1b8728b376214112bed4fb6d916713ec"
format_stamp: "Formatted at 2026-09-28 14:36:44 on dist-test-slave-06lh"
server_key: "4cabddd1d637ffec31b1e3557a20e2b3"
server_key_iv: "dabf94b79f32d066ae6916c9965f2479"
server_key_version: "encryptionkey@0"
I20260928 14:36:44.574384 15478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:36:44.577081 15492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:44.578406 15478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:44.578585 15478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "1b8728b376214112bed4fb6d916713ec"
format_stamp: "Formatted at 2026-09-28 14:36:44 on dist-test-slave-06lh"
server_key: "4cabddd1d637ffec31b1e3557a20e2b3"
server_key_iv: "dabf94b79f32d066ae6916c9965f2479"
server_key_version: "encryptionkey@0"
I20260928 14:36:44.578713 15478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:44.601761 15478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:44.602538 15478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:44.602744 15478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:44.609884 15478 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:41241
I20260928 14:36:44.609894 15544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:41241 every 8 connection(s)
I20260928 14:36:44.610883 15478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:44.614301 15545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:44.616530 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 15478
I20260928 14:36:44.616652 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:44.616974 6239 external_mini_cluster.cc:1428] Setting key 6681f7fbfc1dd5c61b9bc97f500ac899
I20260928 14:36:44.621045 15545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec: Bootstrap starting.
I20260928 14:36:44.623421 15545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:44.624163 15545 log.cc:824] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:44.626587 15545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec: No bootstrap required, opened a new log
I20260928 14:36:44.627345 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:44.618522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.6.23.254:41241)
0928 14:36:44.619337 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:44.619356 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:44.619395 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:36:44.620168 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:44.620187 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:44.620657 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:44.621009 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:44.621484 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.622498 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.623485 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.624024 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.624196 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:44.625115 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:44.625121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:44.625139 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:44.625626 (+ 487us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:44.625699 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:44.625729 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:44.625846 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:44.625996 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 14:36:44.626272 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":144,"threads_started":1}
I20260928 14:36:44.629874 15545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8728b376214112bed4fb6d916713ec" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41241 } }
I20260928 14:36:44.630128 15545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:44.630175 15545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8728b376214112bed4fb6d916713ec, State: Initialized, Role: FOLLOWER
I20260928 14:36:44.630580 15545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8728b376214112bed4fb6d916713ec" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41241 } }
I20260928 14:36:44.630697 15545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:44.630735 15545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:44.630810 15545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:44.631778 15545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8728b376214112bed4fb6d916713ec" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41241 } }
I20260928 14:36:44.632071 15545 leader_election.cc:302] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b8728b376214112bed4fb6d916713ec; no voters:
I20260928 14:36:44.632361 15545 leader_election.cc:288] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:44.632529 15550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:44.632762 15550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [term 1 LEADER]: Becoming Leader. State: Replica: 1b8728b376214112bed4fb6d916713ec, State: Running, Role: LEADER
I20260928 14:36:44.633162 15550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8728b376214112bed4fb6d916713ec" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41241 } }
I20260928 14:36:44.633620 15545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:44.634764 15552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b8728b376214112bed4fb6d916713ec. Latest consensus state: current_term: 1 leader_uuid: "1b8728b376214112bed4fb6d916713ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8728b376214112bed4fb6d916713ec" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41241 } } }
I20260928 14:36:44.634917 15552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:44.635324 15559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:44.635761 15551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b8728b376214112bed4fb6d916713ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b8728b376214112bed4fb6d916713ec" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41241 } } }
I20260928 14:36:44.635865 15551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:44.637979 15559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:44.643947 15559 catalog_manager.cc:1347] Generated new cluster ID: 62c73adbade04b9d9c7068ecebcb203f
I20260928 14:36:44.644032 15559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:44.664206 15559 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:44.665544 15559 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:44.671934 15559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b8728b376214112bed4fb6d916713ec: Generated new TSK 0
I20260928 14:36:44.672785 15559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:36:44.677605 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41241
--builtin_ntp_servers=127.6.23.212:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 14:36:44.795173 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:44.795526 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:44.795567 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:44.799703 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:44.799763 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:44.799820 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:44.804705 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:44.805774 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:44.807143 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:44.816951 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:44.816984 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:44.816957 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:44.817060 15569 server_base.cc:1034] running on GCE node
I20260928 14:36:44.817651 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:44.818276 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:44.819429 15569 hybrid_clock.cc:648] HybridClock initialized: now 1790606204819412 us; error 31 us; skew 500 ppm
I20260928 14:36:44.821306 15569 webserver.cc:460] Webserver started at http://127.6.23.193:39657/ using document root <none> and password file <none>
I20260928 14:36:44.821894 15569 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:44.821956 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:44.822187 15569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:44.824355 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "a17545fdb14248849dea9ae46211cc0a"
format_stamp: "Formatted at 2026-09-28 14:36:44 on dist-test-slave-06lh"
server_key: "c2ab6cea465db687bc88bd93edd4e314"
server_key_iv: "5c0b0b862823480360442aa3afbb1262"
server_key_version: "encryptionkey@0"
I20260928 14:36:44.824924 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "a17545fdb14248849dea9ae46211cc0a"
format_stamp: "Formatted at 2026-09-28 14:36:44 on dist-test-slave-06lh"
server_key: "c2ab6cea465db687bc88bd93edd4e314"
server_key_iv: "5c0b0b862823480360442aa3afbb1262"
server_key_version: "encryptionkey@0"
I20260928 14:36:44.828576 15569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:36:44.831054 15583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:44.832335 15569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:44.832463 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "a17545fdb14248849dea9ae46211cc0a"
format_stamp: "Formatted at 2026-09-28 14:36:44 on dist-test-slave-06lh"
server_key: "c2ab6cea465db687bc88bd93edd4e314"
server_key_iv: "5c0b0b862823480360442aa3afbb1262"
server_key_version: "encryptionkey@0"
I20260928 14:36:44.832593 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:44.850577 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:44.851382 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:44.851581 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:44.852164 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:44.853153 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:44.853220 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:44.853288 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:44.853333 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:44.862120 15569 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:36553
I20260928 14:36:44.862198 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:36553 every 8 connection(s)
I20260928 14:36:44.863118 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:44.865074 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 15569
I20260928 14:36:44.865185 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:44.865483 6239 external_mini_cluster.cc:1428] Setting key e88146c06c779cad96a297b9c7fec93e
I20260928 14:36:44.867393 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41241
--builtin_ntp_servers=127.6.23.212:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:36:44.873831 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:44.865060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:54089 (local address 127.6.23.254:41241)
0928 14:36:44.865220 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:44.865226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:44.866454 (+ 1228us) server_negotiation.cc:407] Connection header received
0928 14:36:44.867983 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:44.867989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:44.868058 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:44.868156 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:44.869333 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.869874 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.871729 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.871939 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.872020 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:44.873368 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:44.873371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:44.873374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:44.873386 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:44.873405 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:44.873413 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:44.873470 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:44.873659 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:36:44.873715 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:36:44.875272 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:44.865546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41241 (local address 127.6.23.193:54089)
0928 14:36:44.866297 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:44.866349 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:44.867770 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:44.868315 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:44.868321 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:44.868413 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:44.869159 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:44.869182 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.870002 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.870009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:44.871602 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:44.871618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:44.872111 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:44.872117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:44.872371 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:44.873042 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:44.873044 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:44.873106 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:44.873203 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:44.873497 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:44.873509 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:44.873518 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:36:44.873667 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 14:36:44.874037 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260928 14:36:44.876631 15697 heartbeater.cc:347] Connected to a master server at 127.6.23.254:41241
I20260928 14:36:44.876870 15697 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:44.877404 15697 heartbeater.cc:511] Master 127.6.23.254:41241 requested a full tablet report, sending...
I20260928 14:36:44.878943 15509 ts_manager.cc:194] Registered new tserver with Master: a17545fdb14248849dea9ae46211cc0a (127.6.23.193:36553)
I20260928 14:36:44.880149 15509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.193:54089
W20260928 14:36:44.983546 15700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:44.983836 15700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:44.983902 15700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:44.988044 15700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:44.988125 15700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:44.988214 15700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:44.992990 15700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:44.994143 15700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:44.995640 15700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:45.005396 15708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:45.005374 15705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:45.005586 15700 server_base.cc:1034] running on GCE node
W20260928 14:36:45.005374 15706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:45.006055 15700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:45.006716 15700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:45.007880 15700 hybrid_clock.cc:648] HybridClock initialized: now 1790606205007858 us; error 33 us; skew 500 ppm
I20260928 14:36:45.009801 15700 webserver.cc:460] Webserver started at http://127.6.23.194:44523/ using document root <none> and password file <none>
I20260928 14:36:45.010435 15700 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:45.010504 15700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:45.010720 15700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:45.012888 15700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "a5d198c658fa45b9990bb02a319781cf"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "a948f6e4adbf5ddd3441f32cb099a938"
server_key_iv: "0fac29d2633bb94f0d1e804bd645e71a"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.013469 15700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "a5d198c658fa45b9990bb02a319781cf"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "a948f6e4adbf5ddd3441f32cb099a938"
server_key_iv: "0fac29d2633bb94f0d1e804bd645e71a"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.017328 15700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:45.020124 15714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.021481 15700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:45.021649 15700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "a5d198c658fa45b9990bb02a319781cf"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "a948f6e4adbf5ddd3441f32cb099a938"
server_key_iv: "0fac29d2633bb94f0d1e804bd645e71a"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.021775 15700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:45.039903 15700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:45.040635 15700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:45.040833 15700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:45.041458 15700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:45.042456 15700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:45.042526 15700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.042591 15700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:45.042629 15700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.051292 15700 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:40715
I20260928 14:36:45.051312 15827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:40715 every 8 connection(s)
I20260928 14:36:45.052249 15700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:45.054369 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 15700
I20260928 14:36:45.054510 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:45.054811 6239 external_mini_cluster.cc:1428] Setting key 8362dcce879577f71e6bd9069ab38312
I20260928 14:36:45.057204 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41241
--builtin_ntp_servers=127.6.23.212:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:36:45.063946 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.054041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:52585 (local address 127.6.23.254:41241)
0928 14:36:45.054175 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:45.054181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:45.055340 (+ 1159us) server_negotiation.cc:407] Connection header received
0928 14:36:45.057211 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:45.057218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:45.057282 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:45.057387 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:45.058793 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:45.059703 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:45.061603 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:45.061835 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:45.061926 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:45.063436 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:45.063439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:45.063441 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:45.063455 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:45.063473 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:45.063481 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:45.063543 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:45.063769 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 14:36:45.063827 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:36:45.065267 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.054399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41241 (local address 127.6.23.194:52585)
0928 14:36:45.055159 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:45.055214 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:45.056954 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:45.057717 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:45.057730 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:45.057850 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:45.058592 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:45.058616 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:45.059863 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:45.059866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:45.061447 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:45.061463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:45.062008 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:45.062015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:45.062261 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:45.062951 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:45.062955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:45.063019 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:45.063146 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:45.063571 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:45.063583 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:45.063592 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:36:45.063746 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 14:36:45.064093 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":133,"threads_started":1}
I20260928 14:36:45.066519 15828 heartbeater.cc:347] Connected to a master server at 127.6.23.254:41241
I20260928 14:36:45.066797 15828 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:45.067536 15828 heartbeater.cc:511] Master 127.6.23.254:41241 requested a full tablet report, sending...
I20260928 14:36:45.068704 15509 ts_manager.cc:194] Registered new tserver with Master: a5d198c658fa45b9990bb02a319781cf (127.6.23.194:40715)
I20260928 14:36:45.069283 15509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.194:52585
W20260928 14:36:45.172086 15831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:45.172386 15831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:45.172453 15831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:45.176614 15831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:45.176689 15831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:45.176775 15831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:45.181586 15831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:45.182765 15831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:45.184216 15831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:45.193595 15839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:45.193717 15836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:45.193926 15831 server_base.cc:1034] running on GCE node
W20260928 14:36:45.193585 15837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:45.194336 15831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:45.194926 15831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:45.196081 15831 hybrid_clock.cc:648] HybridClock initialized: now 1790606205196035 us; error 59 us; skew 500 ppm
I20260928 14:36:45.197885 15831 webserver.cc:460] Webserver started at http://127.6.23.195:35781/ using document root <none> and password file <none>
I20260928 14:36:45.198467 15831 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:45.198529 15831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:45.198746 15831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:45.200867 15831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "fbb34293d95845bbbdf55253a5324a40"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "04beb4ac1290e677288a6f534f21fc08"
server_key_iv: "b84544da78c4bbf3d17c74b7f98eb87e"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.201462 15831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "fbb34293d95845bbbdf55253a5324a40"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "04beb4ac1290e677288a6f534f21fc08"
server_key_iv: "b84544da78c4bbf3d17c74b7f98eb87e"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.205421 15831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:45.208256 15845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.209545 15831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:45.209663 15831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "fbb34293d95845bbbdf55253a5324a40"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "04beb4ac1290e677288a6f534f21fc08"
server_key_iv: "b84544da78c4bbf3d17c74b7f98eb87e"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.209779 15831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:45.245177 15831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:45.245905 15831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:45.246098 15831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:45.246714 15831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:45.247764 15831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:45.247814 15831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.247880 15831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:45.247911 15831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.256723 15831 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:34143
I20260928 14:36:45.256745 15959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:34143 every 8 connection(s)
I20260928 14:36:45.257680 15831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:45.264807 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 15831
I20260928 14:36:45.264935 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:45.265290 6239 external_mini_cluster.cc:1428] Setting key 2e949e8638bacc5d02a04579650bd622
I20260928 14:36:45.267570 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.259576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:57469 (local address 127.6.23.254:41241)
0928 14:36:45.259715 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:45.259718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:45.260819 (+ 1101us) server_negotiation.cc:407] Connection header received
0928 14:36:45.262169 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:45.262171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:45.262217 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:45.262292 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:45.263187 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:45.263869 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:45.265404 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:45.265674 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:45.265790 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:45.267028 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:45.267031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:45.267034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:45.267047 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:45.267071 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:45.267080 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:45.267160 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:45.267342 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 14:36:45.267425 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:36:45.268469 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.259982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41241 (local address 127.6.23.195:57469)
0928 14:36:45.260672 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:45.260713 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:45.261984 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:45.262485 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:45.262491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:45.262576 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:45.263067 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:45.263079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:45.264004 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:45.264007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:45.265266 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:45.265278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:45.265863 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:45.265868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:45.266063 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:45.266777 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:45.266778 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:45.266821 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:45.266889 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:45.267162 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:45.267167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:45.267172 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:36:45.267336 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 14:36:45.267608 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260928 14:36:45.269484 15960 heartbeater.cc:347] Connected to a master server at 127.6.23.254:41241
I20260928 14:36:45.269691 15960 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:45.270164 15960 heartbeater.cc:511] Master 127.6.23.254:41241 requested a full tablet report, sending...
I20260928 14:36:45.271102 15509 ts_manager.cc:194] Registered new tserver with Master: fbb34293d95845bbbdf55253a5324a40 (127.6.23.195:34143)
I20260928 14:36:45.271684 15509 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.195:57469
I20260928 14:36:45.279119 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:36:45.281872 15548 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:45.281039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.6.23.254:41241)
0928 14:36:45.281162 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:45.281166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:45.281334 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:36:45.281442 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:45.281445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:45.281485 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:45.281553 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:45.281765 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35032: BlockingRecv error: recv got EOF from 127.0.0.1:35032 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:36:45.282013 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 15569
I20260928 14:36:45.287672 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 15700
I20260928 14:36:45.292850 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 15831
I20260928 14:36:45.297979 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 15478
2026-09-28T14:36:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (907 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T14:36:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:45Z Disabled control of system clock
I20260928 14:36:45.327389 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:41449
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:41449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:36:45.441612 15971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:45.441897 15971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:45.441936 15971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:45.445998 15971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:45.446079 15971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:45.446095 15971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:45.446113 15971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:45.446130 15971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:45.451114 15971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:41449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:41449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:45.452392 15971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:45.453850 15971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:36:45.462788 15971 server_base.cc:1034] running on GCE node
W20260928 14:36:45.462747 15976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:45.462747 15979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:45.462751 15977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:45.463655 15971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:45.464857 15971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:45.466029 15971 hybrid_clock.cc:648] HybridClock initialized: now 1790606205466012 us; error 30 us; skew 500 ppm
I20260928 14:36:45.467957 15971 webserver.cc:460] Webserver started at http://127.6.23.254:33171/ using document root <none> and password file <none>
I20260928 14:36:45.468521 15971 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:45.468582 15971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:45.468750 15971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:45.470808 15971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "7c4675cf6a538ab543dd362418826e49"
server_key_iv: "5aa6fa0aaeaad1f762e023a7859e66a2"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.471417 15971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "7c4675cf6a538ab543dd362418826e49"
server_key_iv: "5aa6fa0aaeaad1f762e023a7859e66a2"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.475108 15971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:45.477638 15985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.478946 15971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:45.479065 15971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "7c4675cf6a538ab543dd362418826e49"
server_key_iv: "5aa6fa0aaeaad1f762e023a7859e66a2"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.479161 15971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:45.483902 15971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:45.484501 15971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:45.484645 15971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:45.491762 15971 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:41449
I20260928 14:36:45.491816 16037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:41449 every 8 connection(s)
I20260928 14:36:45.492766 15971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:45.493058 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 15971
I20260928 14:36:45.493166 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:45.493515 6239 external_mini_cluster.cc:1428] Setting key 566c5fe54079a09f69f71c0e32a84463
I20260928 14:36:45.496083 16038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:45.500190 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.494977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.6.23.254:41449)
0928 14:36:45.496160 (+ 1183us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:45.496180 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:45.496222 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:36:45.497091 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:45.497115 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:45.497549 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:45.497913 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:45.498635 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:45.498640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:45.498655 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:45.498665 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:45.498728 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:45.498754 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:45.498894 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:45.498962 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:36:45.499256 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":604,"threads_started":1}
I20260928 14:36:45.503032 16038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5: Bootstrap starting.
I20260928 14:36:45.505215 16038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:45.505980 16038 log.cc:824] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:45.508234 16038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5: No bootstrap required, opened a new log
I20260928 14:36:45.512077 16038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41449 } }
I20260928 14:36:45.512331 16038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:45.512365 16038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c3d5b9ae9c4c0aae520298d5ba2bd5, State: Initialized, Role: FOLLOWER
I20260928 14:36:45.512785 16038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41449 } }
I20260928 14:36:45.512912 16038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:45.512959 16038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:45.513037 16038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:45.513944 16038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41449 } }
I20260928 14:36:45.514286 16038 leader_election.cc:302] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38c3d5b9ae9c4c0aae520298d5ba2bd5; no voters:
I20260928 14:36:45.514569 16038 leader_election.cc:288] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:45.514719 16043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:45.514973 16043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [term 1 LEADER]: Becoming Leader. State: Replica: 38c3d5b9ae9c4c0aae520298d5ba2bd5, State: Running, Role: LEADER
I20260928 14:36:45.515383 16043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41449 } }
I20260928 14:36:45.516000 16038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:45.517205 16045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38c3d5b9ae9c4c0aae520298d5ba2bd5. Latest consensus state: current_term: 1 leader_uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41449 } } }
I20260928 14:36:45.517347 16045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:45.517614 16044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38c3d5b9ae9c4c0aae520298d5ba2bd5" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41449 } } }
I20260928 14:36:45.517758 16044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:45.518558 16052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:45.521241 16052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:45.526584 16052 catalog_manager.cc:1347] Generated new cluster ID: be700a77728a43589db094760ba4e758
I20260928 14:36:45.526661 16052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:45.545681 16052 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:45.546070 16052 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:45.556788 16052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38c3d5b9ae9c4c0aae520298d5ba2bd5: Generated new TSK 0
I20260928 14:36:45.557590 16052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:36:45.571677 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41449
--builtin_ntp_servers=127.6.23.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:36:45.702838 16062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:45.703127 16062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:45.703168 16062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:45.707281 16062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:45.707381 16062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:45.707439 16062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:45.712329 16062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:45.713428 16062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:45.714798 16062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:45.724812 16070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:45.724781 16067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:45.724786 16068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:45.724893 16062 server_base.cc:1034] running on GCE node
I20260928 14:36:45.725534 16062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:45.726212 16062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:45.727398 16062 hybrid_clock.cc:648] HybridClock initialized: now 1790606205727386 us; error 24 us; skew 500 ppm
I20260928 14:36:45.729317 16062 webserver.cc:460] Webserver started at http://127.6.23.193:41799/ using document root <none> and password file <none>
I20260928 14:36:45.729949 16062 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:45.730044 16062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:45.730278 16062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:45.732435 16062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "943b2a21eead4ccb80fc1555d5a1ce25"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "97f713192509eb8d9de5ad16ddfe21c9"
server_key_iv: "67fe8c8889da80f32f3d71ff40d34b49"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.733011 16062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "943b2a21eead4ccb80fc1555d5a1ce25"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "97f713192509eb8d9de5ad16ddfe21c9"
server_key_iv: "67fe8c8889da80f32f3d71ff40d34b49"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.736846 16062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:45.739538 16076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.740790 16062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:45.740931 16062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "943b2a21eead4ccb80fc1555d5a1ce25"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "97f713192509eb8d9de5ad16ddfe21c9"
server_key_iv: "67fe8c8889da80f32f3d71ff40d34b49"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.741063 16062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:45.745575 16062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:45.746196 16062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:45.746371 16062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:45.746901 16062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:45.747889 16062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:45.747964 16062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.748049 16062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:45.748092 16062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.757831 16062 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:37319
I20260928 14:36:45.757880 16189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:37319 every 8 connection(s)
I20260928 14:36:45.758816 16062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:45.759061 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 16062
I20260928 14:36:45.759166 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:45.759550 6239 external_mini_cluster.cc:1428] Setting key bddd39330f23c1a7b7cf873cf7d40be3
I20260928 14:36:45.761756 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41449
--builtin_ntp_servers=127.6.23.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:36:45.766561 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.761519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:53735 (local address 127.6.23.254:41449)
0928 14:36:45.761666 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:45.761670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:45.763114 (+ 1444us) server_negotiation.cc:407] Connection header received
0928 14:36:45.764547 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:45.764553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:45.764603 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:45.764697 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:45.765954 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:45.765961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:45.765966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:45.765984 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:45.766014 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:45.766027 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:45.766143 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:45.766303 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 14:36:45.766360 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:36:45.767551 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.762093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41449 (local address 127.6.23.193:53735)
0928 14:36:45.762944 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:45.763009 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:45.764330 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:45.764908 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:45.764916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:45.765013 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:45.765610 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:45.765612 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:45.765675 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:45.765775 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:45.766145 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:45.766154 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:45.766160 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:36:45.766317 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 14:36:45.766530 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260928 14:36:45.768903 16190 heartbeater.cc:347] Connected to a master server at 127.6.23.254:41449
I20260928 14:36:45.769169 16190 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:45.769603 16190 heartbeater.cc:511] Master 127.6.23.254:41449 requested a full tablet report, sending...
I20260928 14:36:45.771421 16002 ts_manager.cc:194] Registered new tserver with Master: 943b2a21eead4ccb80fc1555d5a1ce25 (127.6.23.193:37319)
W20260928 14:36:45.876590 16193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:45.876865 16193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:45.876905 16193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:45.881049 16193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:45.881117 16193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:45.881175 16193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:45.886176 16193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:45.887348 16193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:45.888818 16193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:36:45.898778 16193 server_base.cc:1034] running on GCE node
W20260928 14:36:45.898737 16198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:45.898747 16201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:45.898741 16199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:45.899516 16193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:45.900187 16193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:45.901345 16193 hybrid_clock.cc:648] HybridClock initialized: now 1790606205901302 us; error 58 us; skew 500 ppm
I20260928 14:36:45.903357 16193 webserver.cc:460] Webserver started at http://127.6.23.194:43931/ using document root <none> and password file <none>
I20260928 14:36:45.903955 16193 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:45.904052 16193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:45.904292 16193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:45.906422 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "c5bc4e38b92b41a38b89fe6f449449dc"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "678241370b4bff66e29355712b4844cd"
server_key_iv: "131ec7602cab73b4c4ebfb31ff3cdd33"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.906993 16193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "c5bc4e38b92b41a38b89fe6f449449dc"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "678241370b4bff66e29355712b4844cd"
server_key_iv: "131ec7602cab73b4c4ebfb31ff3cdd33"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.910766 16193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:36:45.913456 16207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.914736 16193 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:36:45.914875 16193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "c5bc4e38b92b41a38b89fe6f449449dc"
format_stamp: "Formatted at 2026-09-28 14:36:45 on dist-test-slave-06lh"
server_key: "678241370b4bff66e29355712b4844cd"
server_key_iv: "131ec7602cab73b4c4ebfb31ff3cdd33"
server_key_version: "encryptionkey@0"
I20260928 14:36:45.914999 16193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:45.920065 16193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:45.920697 16193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:45.920878 16193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:45.921435 16193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:45.922387 16193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:45.922464 16193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.922529 16193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:45.922575 16193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:45.931391 16193 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:34257
I20260928 14:36:45.931414 16320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:34257 every 8 connection(s)
I20260928 14:36:45.932375 16193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:45.938553 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 16193
I20260928 14:36:45.938712 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:45.938877 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.934239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:49133 (local address 127.6.23.254:41449)
0928 14:36:45.934393 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:45.934398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:45.935801 (+ 1403us) server_negotiation.cc:407] Connection header received
0928 14:36:45.937112 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:45.937114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:45.937154 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:45.937229 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:45.938354 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:45.938358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:45.938363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:45.938381 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:45.938408 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:45.938422 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:45.938557 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:45.938681 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 14:36:45.938737 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:36:45.939061 6239 external_mini_cluster.cc:1428] Setting key 4da86b1d2161d54cc8b97f5b01626ee7
I20260928 14:36:45.939931 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:45.934690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41449 (local address 127.6.23.194:49133)
0928 14:36:45.935662 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:45.935706 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:45.936906 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:45.937399 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:45.937406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:45.937498 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:45.938049 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:45.938051 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:45.938093 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:45.938162 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:45.938536 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:45.938543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:45.938548 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:36:45.938716 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 14:36:45.938982 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260928 14:36:45.940994 16321 heartbeater.cc:347] Connected to a master server at 127.6.23.254:41449
I20260928 14:36:45.941241 16321 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:45.941236 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41449
--builtin_ntp_servers=127.6.23.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:36:45.941725 16321 heartbeater.cc:511] Master 127.6.23.254:41449 requested a full tablet report, sending...
I20260928 14:36:45.942866 16002 ts_manager.cc:194] Registered new tserver with Master: c5bc4e38b92b41a38b89fe6f449449dc (127.6.23.194:34257)
W20260928 14:36:46.054998 16324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:46.055353 16324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:46.055442 16324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:46.059871 16324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:46.059947 16324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:46.060037 16324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:46.064920 16324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:41449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:46.066114 16324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:46.067649 16324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:46.077466 16332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:46.077473 16330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:46.077543 16329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:46.077641 16324 server_base.cc:1034] running on GCE node
I20260928 14:36:46.078208 16324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:46.078898 16324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:46.080058 16324 hybrid_clock.cc:648] HybridClock initialized: now 1790606206080013 us; error 59 us; skew 500 ppm
I20260928 14:36:46.082006 16324 webserver.cc:460] Webserver started at http://127.6.23.195:34583/ using document root <none> and password file <none>
I20260928 14:36:46.082638 16324 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:46.082701 16324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:46.082916 16324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:46.085090 16324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "92d86379bad44ab0b6e08737de81eab8"
format_stamp: "Formatted at 2026-09-28 14:36:46 on dist-test-slave-06lh"
server_key: "868c27a7c087eee12fdbb4b326620bf4"
server_key_iv: "b06d59c1edddaf16db66e4c20aad57a9"
server_key_version: "encryptionkey@0"
I20260928 14:36:46.085656 16324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "92d86379bad44ab0b6e08737de81eab8"
format_stamp: "Formatted at 2026-09-28 14:36:46 on dist-test-slave-06lh"
server_key: "868c27a7c087eee12fdbb4b326620bf4"
server_key_iv: "b06d59c1edddaf16db66e4c20aad57a9"
server_key_version: "encryptionkey@0"
I20260928 14:36:46.089488 16324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:46.092056 16338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:46.093281 16324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:36:46.093402 16324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "92d86379bad44ab0b6e08737de81eab8"
format_stamp: "Formatted at 2026-09-28 14:36:46 on dist-test-slave-06lh"
server_key: "868c27a7c087eee12fdbb4b326620bf4"
server_key_iv: "b06d59c1edddaf16db66e4c20aad57a9"
server_key_version: "encryptionkey@0"
I20260928 14:36:46.093516 16324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:46.098078 16324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:46.098744 16324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:46.098919 16324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:46.099503 16324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:46.100430 16324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:46.100485 16324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:46.100551 16324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:46.100587 16324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:46.109135 16324 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:33971
I20260928 14:36:46.109157 16451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:33971 every 8 connection(s)
I20260928 14:36:46.110057 16324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:46.116009 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:46.111956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:35835 (local address 127.6.23.254:41449)
0928 14:36:46.112095 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:46.112099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:46.113160 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 14:36:46.114420 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:46.114423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:46.114467 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:46.114541 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:46.115632 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:46.115635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:46.115637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:36:46.115648 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:46.115667 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:36:46.115675 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:46.115753 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:46.115858 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 14:36:46.115895 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:36:46.116946 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:46.112363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41449 (local address 127.6.23.195:35835)
0928 14:36:46.113025 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:46.113066 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:46.114236 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:46.114699 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:46.114705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:46.114793 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:46.115360 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:36:46.115362 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:46.115403 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:36:46.115494 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:46.115747 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:46.115753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:46.115757 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:36:46.115883 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:36:46.116080 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
I20260928 14:36:46.118076 16452 heartbeater.cc:347] Connected to a master server at 127.6.23.254:41449
I20260928 14:36:46.118346 16452 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:46.118777 16452 heartbeater.cc:511] Master 127.6.23.254:41449 requested a full tablet report, sending...
I20260928 14:36:46.118969 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 16324
I20260928 14:36:46.119074 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:46.119437 6239 external_mini_cluster.cc:1428] Setting key aca60d8deaadc4cb05f19e990c4821de
I20260928 14:36:46.120011 16002 ts_manager.cc:194] Registered new tserver with Master: 92d86379bad44ab0b6e08737de81eab8 (127.6.23.195:33971)
I20260928 14:36:46.121405 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:36:46.124130 16041 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:36:46.123398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.6.23.254:41449)
0928 14:36:46.123550 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:46.123553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:46.123685 (+ 132us) server_negotiation.cc:407] Connection header received
0928 14:36:46.123773 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:46.123775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:46.123806 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:46.123868 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:46.124026 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54002: BlockingRecv error: recv got EOF from 127.0.0.1:54002 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:36:46.124320 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 16062
I20260928 14:36:46.130118 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 16193
I20260928 14:36:46.135596 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 16324
I20260928 14:36:46.140839 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 15971
2026-09-28T14:36:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (842 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:46 dist-test-slave-06lh krb5kdc[16462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:46 dist-test-slave-06lh krb5kdc[16462](info): set up 2 sockets
Sep 28 14:36:46 dist-test-slave-06lh krb5kdc[16462](info): commencing operation
krb5kdc: starting...
W20260928 14:36:47.180563 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:47 dist-test-slave-06lh krb5kdc[16462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:36:47.306723 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45853
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:47.421263 16475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:47.421598 16475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:47.421644 16475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:47.425953 16475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:36:47.426028 16475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:47.426047 16475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:47.426065 16475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:36:47.426081 16475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:36:47.432482 16475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:47.433601 16475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:47.434634 16475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:47.443521 16481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:47.443538 16480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:47.443521 16483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:47.443816 16475 server_base.cc:1034] running on GCE node
I20260928 14:36:47.444573 16475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:47.445796 16475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:47.447016 16475 hybrid_clock.cc:648] HybridClock initialized: now 1790606207446996 us; error 33 us; skew 500 ppm
Sep 28 14:36:47 dist-test-slave-06lh krb5kdc[16462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:47.450237 16475 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:36:47.451145 16475 webserver.cc:460] Webserver started at http://127.6.23.254:33173/ using document root <none> and password file <none>
I20260928 14:36:47.451751 16475 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:47.451815 16475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:47.451985 16475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:47.454126 16475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "9b832e1978c64a0f881868cd07d5c8bf"
format_stamp: "Formatted at 2026-09-28 14:36:47 on dist-test-slave-06lh"
server_key: "aa018e425052b4c2901e0f5487a0fb57"
server_key_iv: "334473908a27f2c14bb764c6e8fa6e92"
server_key_version: "encryptionkey@0"
I20260928 14:36:47.454699 16475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "9b832e1978c64a0f881868cd07d5c8bf"
format_stamp: "Formatted at 2026-09-28 14:36:47 on dist-test-slave-06lh"
server_key: "aa018e425052b4c2901e0f5487a0fb57"
server_key_iv: "334473908a27f2c14bb764c6e8fa6e92"
server_key_version: "encryptionkey@0"
I20260928 14:36:47.458813 16475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:36:47.461519 16490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:47.462911 16475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:47.463032 16475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "9b832e1978c64a0f881868cd07d5c8bf"
format_stamp: "Formatted at 2026-09-28 14:36:47 on dist-test-slave-06lh"
server_key: "aa018e425052b4c2901e0f5487a0fb57"
server_key_iv: "334473908a27f2c14bb764c6e8fa6e92"
server_key_version: "encryptionkey@0"
I20260928 14:36:47.463136 16475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:47.478413 16475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:47.481982 16475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:47.482203 16475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:47.489102 16475 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:45853
I20260928 14:36:47.489436 16542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:45853 every 8 connection(s)
I20260928 14:36:47.490160 16475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:36:47.493132 16543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:47.493669 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 16475
I20260928 14:36:47.493789 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:36:47.494118 6239 external_mini_cluster.cc:1428] Setting key 802ba4687a789ee8ba34257ead8ad17d
I20260928 14:36:47.500923 16543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf: Bootstrap starting.
I20260928 14:36:47.503831 16543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf: Neither blocks nor log segments found. Creating new log.
Sep 28 14:36:47 dist-test-slave-06lh krb5kdc[16462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:47.504789 16543 log.cc:824] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:47.506893 16543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf: No bootstrap required, opened a new log
I20260928 14:36:47.509657 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:47.495587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.6.23.254:45853)
0928 14:36:47.496042 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:47.496062 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:47.496108 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:36:47.496882 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:47.496907 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:47.497355 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:47.497762 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:47.498437 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:47.500014 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:47.500909 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:47.501596 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:47.501746 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:47.504656 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:47.504662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:47.504676 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:47.504704 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:47.506501 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:47.507072 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:47.507078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:47.507083 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:47.507205 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:47.507589 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:47.507592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:47.507593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:47.508038 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:47.508197 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:47.508539 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 14:36:47.508901 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":142,"threads_started":1}
I20260928 14:36:47.511003 16543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b832e1978c64a0f881868cd07d5c8bf" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45853 } }
I20260928 14:36:47.511278 16543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:47.511353 16543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b832e1978c64a0f881868cd07d5c8bf, State: Initialized, Role: FOLLOWER
I20260928 14:36:47.511852 16543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b832e1978c64a0f881868cd07d5c8bf" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45853 } }
I20260928 14:36:47.512029 16543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:36:47.512122 16543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:36:47.512226 16543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:47.513167 16543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b832e1978c64a0f881868cd07d5c8bf" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45853 } }
I20260928 14:36:47.513481 16543 leader_election.cc:302] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b832e1978c64a0f881868cd07d5c8bf; no voters:
I20260928 14:36:47.513741 16543 leader_election.cc:288] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:36:47.513942 16548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:47.514206 16548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [term 1 LEADER]: Becoming Leader. State: Replica: 9b832e1978c64a0f881868cd07d5c8bf, State: Running, Role: LEADER
I20260928 14:36:47.514549 16548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b832e1978c64a0f881868cd07d5c8bf" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45853 } }
I20260928 14:36:47.515043 16543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:36:47.516252 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b832e1978c64a0f881868cd07d5c8bf. Latest consensus state: current_term: 1 leader_uuid: "9b832e1978c64a0f881868cd07d5c8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b832e1978c64a0f881868cd07d5c8bf" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45853 } } }
I20260928 14:36:47.516423 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:47.516878 16557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:36:47.516867 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b832e1978c64a0f881868cd07d5c8bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b832e1978c64a0f881868cd07d5c8bf" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45853 } } }
I20260928 14:36:47.517062 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf [sys.catalog]: This master's current role is: LEADER
I20260928 14:36:47.520004 16557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:36:47.526203 16557 catalog_manager.cc:1347] Generated new cluster ID: b619f4600a7c4c868e560cf7988a905d
I20260928 14:36:47.526371 16557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:36:47.548781 16557 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:36:47.550163 16557 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:36:47.556322 16557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf: Generated new TSK 0
I20260928 14:36:47.557237 16557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:36:47.592301 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45853
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:47.710427 16569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:47.710773 16569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:47.710847 16569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:47.715164 16569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:47.715296 16569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:47.715444 16569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:36:47.721284 16569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:47.722458 16569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:47.723568 16569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:47.733405 16574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:47.733413 16575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:47.733621 16577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:47.733479 16569 server_base.cc:1034] running on GCE node
I20260928 14:36:47.734155 16569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:47.734874 16569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:47.736040 16569 hybrid_clock.cc:648] HybridClock initialized: now 1790606207736000 us; error 57 us; skew 500 ppm
Sep 28 14:36:47 dist-test-slave-06lh krb5kdc[16462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:47.738924 16569 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:36:47.739872 16569 webserver.cc:460] Webserver started at http://127.6.23.193:43977/ using document root <none> and password file <none>
I20260928 14:36:47.740422 16569 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:47.740485 16569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:47.740653 16569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:47.742676 16569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "20bb78871cb94fd7b54c5a357bdd95f4"
format_stamp: "Formatted at 2026-09-28 14:36:47 on dist-test-slave-06lh"
server_key: "5a2b4ecfa96bb048dd796d84a682ed7e"
server_key_iv: "5720a83d474e06c2693744ce6d69457a"
server_key_version: "encryptionkey@0"
I20260928 14:36:47.743192 16569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "20bb78871cb94fd7b54c5a357bdd95f4"
format_stamp: "Formatted at 2026-09-28 14:36:47 on dist-test-slave-06lh"
server_key: "5a2b4ecfa96bb048dd796d84a682ed7e"
server_key_iv: "5720a83d474e06c2693744ce6d69457a"
server_key_version: "encryptionkey@0"
I20260928 14:36:47.747028 16569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:36:47.749676 16584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:47.751137 16569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:36:47.751318 16569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "20bb78871cb94fd7b54c5a357bdd95f4"
format_stamp: "Formatted at 2026-09-28 14:36:47 on dist-test-slave-06lh"
server_key: "5a2b4ecfa96bb048dd796d84a682ed7e"
server_key_iv: "5720a83d474e06c2693744ce6d69457a"
server_key_version: "encryptionkey@0"
I20260928 14:36:47.751471 16569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:47.790180 16569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:47.793682 16569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:47.793915 16569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:47.794562 16569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:47.795480 16569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:47.795545 16569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:47.795619 16569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:47.795658 16569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:47.805270 16569 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:33027
I20260928 14:36:47.805290 16697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:33027 every 8 connection(s)
I20260928 14:36:47.806285 16569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:36:47.810009 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 16569
I20260928 14:36:47.810140 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:36:47.810453 6239 external_mini_cluster.cc:1428] Setting key 700164e583419a62f75347ae8ca8c754
Sep 28 14:36:47 dist-test-slave-06lh krb5kdc[16462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:36:47.820829 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:47.808277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:45663 (local address 127.6.23.254:45853)
0928 14:36:47.808437 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:47.808441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:47.809266 (+ 825us) server_negotiation.cc:407] Connection header received
0928 14:36:47.810808 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:47.810811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:47.810877 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:47.810980 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:47.812297 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:47.812833 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:47.814048 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:47.814286 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:47.814390 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:47.817162 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:47.817165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:47.817167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:47.817193 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:47.818616 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:47.819160 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:47.819166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:47.819169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:47.819298 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:47.819655 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:47.819658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:47.819660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:47.819899 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:47.819997 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:47.820524 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 14:36:47.820659 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:36:47.821842 16700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:47.808633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45853 (local address 127.6.23.193:45663)
0928 14:36:47.809103 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:47.809150 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:47.810571 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:47.811171 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:47.811178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:47.811615 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:47.812142 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:47.812157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:47.812969 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:47.812978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:47.813905 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:47.813923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:47.814450 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:47.814455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:47.814682 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:47.815357 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:47.815374 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:47.816976 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:47.818747 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:47.818754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:47.818772 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:47.819054 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:47.819395 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:47.819398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:47.819400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:47.819553 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:47.820008 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:47.820016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:47.820285 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 14:36:47.820485 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 14:36:47.820917 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:36:47.823328 16698 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45853
I20260928 14:36:47.823628 16698 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:47.824239 16698 heartbeater.cc:511] Master 127.6.23.254:45853 requested a full tablet report, sending...
I20260928 14:36:47.826015 16503 ts_manager.cc:194] Registered new tserver with Master: 20bb78871cb94fd7b54c5a357bdd95f4 (127.6.23.193:33027)
I20260928 14:36:47.827183 16503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:45663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:36:47.842845 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45853
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:47.972429 16703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:47.973261 16703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:47.973366 16703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:47.977778 16703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:47.977859 16703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:47.977954 16703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:36:47.983772 16703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:47.984994 16703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:47.986112 16703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:36:47.998705 16711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:47.998705 16709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:47.998704 16708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:47.999074 16703 server_base.cc:1034] running on GCE node
I20260928 14:36:47.999557 16703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:48.000272 16703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:48.001482 16703 hybrid_clock.cc:648] HybridClock initialized: now 1790606208001475 us; error 50 us; skew 500 ppm
Sep 28 14:36:48 dist-test-slave-06lh krb5kdc[16462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:48.004550 16703 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:36:48.005487 16703 webserver.cc:460] Webserver started at http://127.6.23.194:43825/ using document root <none> and password file <none>
I20260928 14:36:48.006117 16703 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:48.006211 16703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:48.006445 16703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:48.008698 16703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "dafb36a1f112442295e5da0f0febdebf"
format_stamp: "Formatted at 2026-09-28 14:36:48 on dist-test-slave-06lh"
server_key: "d69e103e51969870f9ec19b4779f96f1"
server_key_iv: "bdbbef7c177626701e8673335c563789"
server_key_version: "encryptionkey@0"
I20260928 14:36:48.009341 16703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "dafb36a1f112442295e5da0f0febdebf"
format_stamp: "Formatted at 2026-09-28 14:36:48 on dist-test-slave-06lh"
server_key: "d69e103e51969870f9ec19b4779f96f1"
server_key_iv: "bdbbef7c177626701e8673335c563789"
server_key_version: "encryptionkey@0"
I20260928 14:36:48.013404 16703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:36:48.016474 16718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:48.017912 16703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:36:48.018052 16703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "dafb36a1f112442295e5da0f0febdebf"
format_stamp: "Formatted at 2026-09-28 14:36:48 on dist-test-slave-06lh"
server_key: "d69e103e51969870f9ec19b4779f96f1"
server_key_iv: "bdbbef7c177626701e8673335c563789"
server_key_version: "encryptionkey@0"
I20260928 14:36:48.018175 16703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:48.036746 16703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:48.040357 16703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:48.040565 16703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:48.041191 16703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:48.042068 16703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:48.042115 16703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:48.042183 16703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:48.042222 16703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:48.050951 16703 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:41315
I20260928 14:36:48.051028 16831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:41315 every 8 connection(s)
I20260928 14:36:48.051939 16703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:36:48.060159 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 16703
I20260928 14:36:48.060292 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:36:48.060621 6239 external_mini_cluster.cc:1428] Setting key fcb43a147bbcb25ad3c6339e5db5bcdb
Sep 28 14:36:48 dist-test-slave-06lh krb5kdc[16462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:48.069646 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.053878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:45305 (local address 127.6.23.254:45853)
0928 14:36:48.054023 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.054027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.055445 (+ 1418us) server_negotiation.cc:407] Connection header received
0928 14:36:48.056592 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.056595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.056652 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.056736 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:48.058142 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.058675 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.059919 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.060279 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.060393 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.063042 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:48.063049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:48.063054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:48.063088 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:48.067112 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:48.067916 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:48.067919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:48.067921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:48.068012 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:48.068422 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:48.068424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:48.068425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:48.068616 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:48.068695 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.069405 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.069515 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:36:48.070935 16834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.054260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45853 (local address 127.6.23.194:45305)
0928 14:36:48.055280 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:48.055345 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:48.056351 (+ 1006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:48.056924 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:48.056931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:48.057413 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:48.057963 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.057984 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.058792 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.058794 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.059797 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.059807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.060543 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.060546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.060734 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.061360 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:48.061378 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:48.062873 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:48.067278 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:48.067290 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:48.067322 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:48.067813 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:48.068104 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:48.068109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:48.068112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:48.068323 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:48.068727 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:48.068738 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:48.069128 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 14:36:48.069371 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 14:36:48.069817 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":616,"threads_started":1}
I20260928 14:36:48.072206 16832 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45853
I20260928 14:36:48.072499 16832 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:36:48.073137 16832 heartbeater.cc:511] Master 127.6.23.254:45853 requested a full tablet report, sending...
I20260928 14:36:48.074419 16503 ts_manager.cc:194] Registered new tserver with Master: dafb36a1f112442295e5da0f0febdebf (127.6.23.194:41315)
I20260928 14:36:48.075105 16503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:45305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:36:48.091374 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45853
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:44987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:36:48.207456 16837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:36:48.207746 16837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:36:48.207790 16837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:36:48.211930 16837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:36:48.212003 16837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:36:48.212064 16837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:36:48.217734 16837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:44987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:36:48.218847 16837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:36:48.219985 16837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:36:48.230135 16837 server_base.cc:1034] running on GCE node
W20260928 14:36:48.230060 16843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:48.230077 16842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:36:48.230044 16845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:36:48.230844 16837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:36:48.231581 16837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:36:48.232764 16837 hybrid_clock.cc:648] HybridClock initialized: now 1790606208232742 us; error 36 us; skew 500 ppm
Sep 28 14:36:48 dist-test-slave-06lh krb5kdc[16462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:36:48.235790 16837 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:36:48.236704 16837 webserver.cc:460] Webserver started at http://127.6.23.195:38869/ using document root <none> and password file <none>
I20260928 14:36:48.237308 16837 fs_manager.cc:359] Metadata directory not provided
I20260928 14:36:48.237408 16837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:36:48.237648 16837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:36:48.239841 16837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "6608d162b49e44cda83e4d50714a834c"
format_stamp: "Formatted at 2026-09-28 14:36:48 on dist-test-slave-06lh"
server_key: "4cfc931fca19d585310caf8c1e90caf3"
server_key_iv: "dfad1bf19963f4b13084e4a95c95a7ef"
server_key_version: "encryptionkey@0"
I20260928 14:36:48.240480 16837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "6608d162b49e44cda83e4d50714a834c"
format_stamp: "Formatted at 2026-09-28 14:36:48 on dist-test-slave-06lh"
server_key: "4cfc931fca19d585310caf8c1e90caf3"
server_key_iv: "dfad1bf19963f4b13084e4a95c95a7ef"
server_key_version: "encryptionkey@0"
I20260928 14:36:48.244313 16837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:36:48.246896 16852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:48.248255 16837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:36:48.248395 16837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "6608d162b49e44cda83e4d50714a834c"
format_stamp: "Formatted at 2026-09-28 14:36:48 on dist-test-slave-06lh"
server_key: "4cfc931fca19d585310caf8c1e90caf3"
server_key_iv: "dfad1bf19963f4b13084e4a95c95a7ef"
server_key_version: "encryptionkey@0"
I20260928 14:36:48.248520 16837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:36:48.266944 16837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:36:48.270618 16837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:36:48.270824 16837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:36:48.271565 16837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:36:48.272481 16837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:36:48.272559 16837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:48.272634 16837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:36:48.272682 16837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:36:48.281854 16837 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:36243
I20260928 14:36:48.281872 16965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:36243 every 8 connection(s)
I20260928 14:36:48.282809 16837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:36:48.288298 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 16837
I20260928 14:36:48.288403 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:36:48.288703 6239 external_mini_cluster.cc:1428] Setting key 66d6b935e033ffaf1b2685a634bae0d9
Sep 28 14:36:48 dist-test-slave-06lh krb5kdc[16462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606208, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:36:48.298344 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.284790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:48129 (local address 127.6.23.254:45853)
0928 14:36:48.284931 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.284934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.285762 (+ 828us) server_negotiation.cc:407] Connection header received
0928 14:36:48.287163 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.287166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.287217 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.287379 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:48.288913 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.289652 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.291387 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.291624 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.291719 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.294530 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:48.294533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:48.294535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:48.294562 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:48.296162 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:48.296758 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:48.296761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:48.296762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:48.296847 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:48.297269 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:48.297271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:48.297272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:48.297472 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:48.297550 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.298098 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.298211 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:36:48.299494 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.285117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45853 (local address 127.6.23.195:48129)
0928 14:36:48.285588 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:48.285635 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:48.286943 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:48.287564 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:48.287571 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:48.287966 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:36:48.288681 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.288700 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.289852 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.289856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.291264 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.291277 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.291776 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.291779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.291948 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.292668 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:36:48.292688 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:36:48.294332 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:36:48.296317 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:48.296324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:48.296345 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:48.296644 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:48.296966 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:36:48.296969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:36:48.296971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:36:48.297145 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:36:48.297570 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:36:48.297576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:36:48.297873 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 14:36:48.298052 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 14:36:48.298365 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
I20260928 14:36:48.300908 16966 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45853
I20260928 14:36:48.301205 16966 heartbeater.cc:464] Registering TS with master...
I20260928 14:36:48.301918 16966 heartbeater.cc:511] Master 127.6.23.254:45853 requested a full tablet report, sending...
I20260928 14:36:48.303210 16507 ts_manager.cc:194] Registered new tserver with Master: 6608d162b49e44cda83e4d50714a834c (127.6.23.195:36243)
I20260928 14:36:48.303891 16507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:48129
I20260928 14:36:48.315848 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:36:48.336585 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.318129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.6.23.254:45853)
0928 14:36:48.318294 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.318302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.318488 (+ 186us) server_negotiation.cc:407] Connection header received
0928 14:36:48.318657 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.318660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.318710 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.318787 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:36:48.319581 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.320144 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.321053 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.321285 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.321390 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.333406 (+ 12016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:36:48.333410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:36:48.333412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:36:48.333449 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:36:48.334740 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:48.335208 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:48.335213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:48.335214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:48.335348 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:36:48.335631 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:36:48.335634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:36:48.335636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:36:48.335886 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:36:48.336026 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.336272 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.336409 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:36:48.339950 16507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:36:48.343106 16507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:36:48.358963 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.353500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.6.23.195:36243)
0928 14:36:48.353863 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.353874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.354022 (+ 148us) server_negotiation.cc:407] Connection header received
0928 14:36:48.354163 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.354167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.354380 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.354545 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:48.354777 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.356332 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.357341 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.358206 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.358320 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.358502 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.358682 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.358787 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":172,"threads_started":1}
I20260928 14:36:48.358978 16977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.353103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:36243 (local address 127.0.0.1:42834)
0928 14:36:48.353925 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:48.353938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:48.354052 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:48.354519 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:48.354522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:48.354536 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:48.354628 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.354633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.356459 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.356462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.357215 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.357224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.358335 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.358338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.358459 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.358511 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 14:36:48.358637 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:36:48.358806 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":40,"thread_start_us":86,"threads_started":1}
I20260928 14:36:48.359689 16979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.352253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:33027 (local address 127.0.0.1:49572)
0928 14:36:48.353391 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:48.353448 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:48.353587 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:48.354250 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:48.354256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:48.354278 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:48.354431 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.354442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.356002 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.356006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.357422 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.357433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.358493 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.358496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.358573 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.358608 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:36:48.358673 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:36:48.358806 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":103,"spinlock_wait_cycles":5760,"thread_start_us":150,"threads_started":1}
I20260928 14:36:48.360096 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.352386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.6.23.193:33027)
0928 14:36:48.353866 (+ 1480us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.353873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.353891 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:36:48.353965 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.353969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.354114 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.354237 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:48.354569 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.355878 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.357546 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.358315 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.358461 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.358583 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.359854 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.359966 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1378,"thread_start_us":112,"threads_started":1}
I20260928 14:36:48.360399 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.352456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:41315 (local address 127.0.0.1:49558)
0928 14:36:48.353396 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:48.353452 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:48.353587 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:48.354386 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:48.354388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:48.354404 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:48.354518 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.354524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.355695 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.355701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.357425 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.357433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.359956 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.359961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.360057 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.360090 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:36:48.360161 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:36:48.360243 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":89,"spinlock_wait_cycles":3072,"thread_start_us":113,"threads_started":1}
I20260928 14:36:48.360826 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.352843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.6.23.194:41315)
0928 14:36:48.353223 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.353236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.353733 (+ 497us) server_negotiation.cc:407] Connection header received
0928 14:36:48.353817 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.353823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.354043 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.354198 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:48.354646 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.355576 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.358674 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.359620 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.359728 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.359894 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.360585 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.360688 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260928 14:36:48.362986 16900 tablet_service.cc:1467] Processing CreateTablet for tablet 1699986d9a004834afdcb721bf31beda (DEFAULT_TABLE table=test-table [id=19ddbc5a0eb4424da9255dbbffb51550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:48.363054 16632 tablet_service.cc:1467] Processing CreateTablet for tablet 1699986d9a004834afdcb721bf31beda (DEFAULT_TABLE table=test-table [id=19ddbc5a0eb4424da9255dbbffb51550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:48.363545 16766 tablet_service.cc:1467] Processing CreateTablet for tablet 1699986d9a004834afdcb721bf31beda (DEFAULT_TABLE table=test-table [id=19ddbc5a0eb4424da9255dbbffb51550]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:36:48.364245 16632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1699986d9a004834afdcb721bf31beda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:48.364262 16900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1699986d9a004834afdcb721bf31beda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:48.364428 16766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1699986d9a004834afdcb721bf31beda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:36:48.371541 16983 tablet_bootstrap.cc:492] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Bootstrap starting.
I20260928 14:36:48.372500 16984 tablet_bootstrap.cc:492] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4: Bootstrap starting.
I20260928 14:36:48.372882 16985 tablet_bootstrap.cc:492] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Bootstrap starting.
I20260928 14:36:48.374508 16983 tablet_bootstrap.cc:654] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:48.374926 16984 tablet_bootstrap.cc:654] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:48.375502 16983 log.cc:824] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:48.375573 16985 tablet_bootstrap.cc:654] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Neither blocks nor log segments found. Creating new log.
I20260928 14:36:48.375880 16984 log.cc:824] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:48.376405 16985 log.cc:824] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:36:48.377825 16983 tablet_bootstrap.cc:492] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: No bootstrap required, opened a new log
I20260928 14:36:48.377831 16984 tablet_bootstrap.cc:492] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4: No bootstrap required, opened a new log
I20260928 14:36:48.378046 16984 ts_tablet_manager.cc:1397] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:36:48.378037 16983 ts_tablet_manager.cc:1397] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:36:48.378273 16985 tablet_bootstrap.cc:492] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: No bootstrap required, opened a new log
I20260928 14:36:48.378443 16985 ts_tablet_manager.cc:1397] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:36:48.381749 16984 raft_consensus.cc:348] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.382066 16984 raft_consensus.cc:374] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:48.382120 16984 raft_consensus.cc:729] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20bb78871cb94fd7b54c5a357bdd95f4, State: Initialized, Role: FOLLOWER
I20260928 14:36:48.382097 16985 raft_consensus.cc:348] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.382097 16983 raft_consensus.cc:348] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.382386 16985 raft_consensus.cc:374] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:48.382450 16983 raft_consensus.cc:374] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:36:48.382457 16985 raft_consensus.cc:729] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafb36a1f112442295e5da0f0febdebf, State: Initialized, Role: FOLLOWER
I20260928 14:36:48.382510 16983 raft_consensus.cc:729] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6608d162b49e44cda83e4d50714a834c, State: Initialized, Role: FOLLOWER
I20260928 14:36:48.382622 16984 consensus_queue.cc:260] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.382951 16983 consensus_queue.cc:260] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.382929 16985 consensus_queue.cc:260] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.383505 16698 heartbeater.cc:503] Master 127.6.23.254:45853 was elected leader, sending a full tablet report...
I20260928 14:36:48.383770 16984 ts_tablet_manager.cc:1428] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:36:48.383947 16985 ts_tablet_manager.cc:1428] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260928 14:36:48.383941 16983 ts_tablet_manager.cc:1428] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:36:48.384280 16832 heartbeater.cc:503] Master 127.6.23.254:45853 was elected leader, sending a full tablet report...
I20260928 14:36:48.384296 16966 heartbeater.cc:503] Master 127.6.23.254:45853 was elected leader, sending a full tablet report...
I20260928 14:36:48.506088 16989 raft_consensus.cc:482] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:36:48.506361 16989 raft_consensus.cc:504] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.507663 16989 leader_election.cc:288] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6608d162b49e44cda83e4d50714a834c (127.6.23.195:36243), dafb36a1f112442295e5da0f0febdebf (127.6.23.194:41315)
I20260928 14:36:48.511904 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.508019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:38079 (local address 127.6.23.195:36243)
0928 14:36:48.508199 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.508204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.508416 (+ 212us) server_negotiation.cc:407] Connection header received
0928 14:36:48.508548 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.508553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.508613 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.508677 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:48.509046 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.509570 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.510781 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.511187 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.511394 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.511465 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.511665 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.511749 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 14:36:48.511909 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.508091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:45607 (local address 127.6.23.194:41315)
0928 14:36:48.508224 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.508228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.508503 (+ 275us) server_negotiation.cc:407] Connection header received
0928 14:36:48.508617 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.508619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.508685 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.508767 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:36:48.509094 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.509588 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.510786 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.511189 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.511390 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.511453 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.511692 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.511779 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:36:48.511951 16993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.507884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:36243 (local address 127.6.23.193:38079)
0928 14:36:48.508312 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:48.508324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:48.508434 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:48.508715 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:48.508721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:48.508741 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:48.508932 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.508939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.509688 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.509692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.510651 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.510661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.511385 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.511390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.511514 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.511562 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:36:48.511625 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:36:48.511762 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":130,"thread_start_us":138,"threads_started":1}
I20260928 14:36:48.512377 16992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.507999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:41315 (local address 127.6.23.193:45607)
0928 14:36:48.508361 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:48.508392 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 14:36:48.508513 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:36:48.508791 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:36:48.508798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:36:48.508820 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:36:48.508961 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.508966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.509719 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.509722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.510652 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:36:48.510663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.511394 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.511398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:36:48.511557 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.511574 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 14:36:48.511664 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:36:48.511770 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":83,"spinlock_wait_cycles":896,"thread_start_us":135,"threads_started":1}
I20260928 14:36:48.512691 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1699986d9a004834afdcb721bf31beda" candidate_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6608d162b49e44cda83e4d50714a834c" is_pre_election: true
I20260928 14:36:48.513090 16920 raft_consensus.cc:2454] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bb78871cb94fd7b54c5a357bdd95f4 in term 0.
I20260928 14:36:48.513118 16786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1699986d9a004834afdcb721bf31beda" candidate_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafb36a1f112442295e5da0f0febdebf" is_pre_election: true
I20260928 14:36:48.513417 16786 raft_consensus.cc:2454] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20bb78871cb94fd7b54c5a357bdd95f4 in term 0.
I20260928 14:36:48.513633 16587 leader_election.cc:302] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bb78871cb94fd7b54c5a357bdd95f4, 6608d162b49e44cda83e4d50714a834c; no voters:
I20260928 14:36:48.513972 16989 raft_consensus.cc:2781] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:36:48.514060 16989 raft_consensus.cc:482] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:36:48.514103 16989 raft_consensus.cc:3037] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:48.515332 16989 raft_consensus.cc:504] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.516062 16989 leader_election.cc:288] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [CANDIDATE]: Term 1 election: Requested vote from peers 6608d162b49e44cda83e4d50714a834c (127.6.23.195:36243), dafb36a1f112442295e5da0f0febdebf (127.6.23.194:41315)
I20260928 14:36:48.516146 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1699986d9a004834afdcb721bf31beda" candidate_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6608d162b49e44cda83e4d50714a834c"
I20260928 14:36:48.516168 16786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1699986d9a004834afdcb721bf31beda" candidate_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dafb36a1f112442295e5da0f0febdebf"
I20260928 14:36:48.516288 16920 raft_consensus.cc:3037] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:48.516319 16786 raft_consensus.cc:3037] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:36:48.517524 16920 raft_consensus.cc:2454] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bb78871cb94fd7b54c5a357bdd95f4 in term 1.
I20260928 14:36:48.517611 16786 raft_consensus.cc:2454] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20bb78871cb94fd7b54c5a357bdd95f4 in term 1.
I20260928 14:36:48.517841 16587 leader_election.cc:302] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20bb78871cb94fd7b54c5a357bdd95f4, 6608d162b49e44cda83e4d50714a834c; no voters:
I20260928 14:36:48.518011 16989 raft_consensus.cc:2781] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:36:48.518229 16989 raft_consensus.cc:686] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [term 1 LEADER]: Becoming Leader. State: Replica: 20bb78871cb94fd7b54c5a357bdd95f4, State: Running, Role: LEADER
I20260928 14:36:48.518585 16989 consensus_queue.cc:237] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } }
I20260928 14:36:48.522149 16507 catalog_manager.cc:5495] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20bb78871cb94fd7b54c5a357bdd95f4 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20bb78871cb94fd7b54c5a357bdd95f4" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 } health_report { overall_health: UNKNOWN } } }
W20260928 14:36:48.533828 16967 tablet.cc:2369] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:48.549932 16978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.546209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.6.23.193:33027)
0928 14:36:48.546345 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:36:48.546349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:36:48.546364 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:36:48.546505 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:36:48.546507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:36:48.546565 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:36:48.546682 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:36:48.546952 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.547538 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.548442 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:36:48.548661 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:36:48.548763 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:36:48.549090 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:36:48.549533 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:36:48.549668 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:36:48.549715 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 14:36:48.549776 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:36:48.553107 16833 tablet.cc:2369] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:48.556694 16786 raft_consensus.cc:1261] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 1 FOLLOWER]: Refusing update from remote peer 20bb78871cb94fd7b54c5a357bdd95f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:48.556694 16920 raft_consensus.cc:1261] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 1 FOLLOWER]: Refusing update from remote peer 20bb78871cb94fd7b54c5a357bdd95f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:36:48.557432 16994 consensus_queue.cc:1035] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dafb36a1f112442295e5da0f0febdebf" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:36:48.557636 16699 tablet.cc:2369] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:36:48.557667 16989 consensus_queue.cc:1035] T 1699986d9a004834afdcb721bf31beda P 20bb78871cb94fd7b54c5a357bdd95f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6608d162b49e44cda83e4d50714a834c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:36:48.567934 16998 mvcc.cc:205] Tried to move back new op lower bound from 7334323030240100352 to 7334323030096068608. Current Snapshot: MvccSnapshot[applied={T|T < 7334323030240100352 or (T in {7334323030240100352})}]
I20260928 14:36:48.574070 16501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:48.574260 16501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:36:48.576707 16501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b832e1978c64a0f881868cd07d5c8bf: Sending DeleteTablet for 3 replicas of tablet 1699986d9a004834afdcb721bf31beda
I20260928 14:36:48.577612 16632 tablet_service.cc:1514] Processing DeleteTablet for tablet 1699986d9a004834afdcb721bf31beda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:48 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:49572
I20260928 14:36:48.577612 16900 tablet_service.cc:1514] Processing DeleteTablet for tablet 1699986d9a004834afdcb721bf31beda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:48 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:42834
I20260928 14:36:48.578258 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 16569
I20260928 14:36:48.578296 16766 tablet_service.cc:1514] Processing DeleteTablet for tablet 1699986d9a004834afdcb721bf31beda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:36:48 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:49558
I20260928 14:36:48.579423 17006 tablet_replica.cc:330] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: stopping tablet replica
I20260928 14:36:48.579639 17008 tablet_replica.cc:330] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: stopping tablet replica
I20260928 14:36:48.579804 17006 raft_consensus.cc:2229] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:48.579967 17008 raft_consensus.cc:2229] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:36:48.580173 17006 raft_consensus.cc:2258] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:48.580264 17008 raft_consensus.cc:2258] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:36:48.581979 17008 ts_tablet_manager.cc:1905] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:48.582868 17006 ts_tablet_manager.cc:1905] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:36:48.584352 17008 ts_tablet_manager.cc:1918] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:48.584465 17008 log.cc:1196] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/1699986d9a004834afdcb721bf31beda
I20260928 14:36:48.584765 17008 ts_tablet_manager.cc:1939] T 1699986d9a004834afdcb721bf31beda P dafb36a1f112442295e5da0f0febdebf: Deleting consensus metadata
I20260928 14:36:48.586071 16492 catalog_manager.cc:4841] TS dafb36a1f112442295e5da0f0febdebf (127.6.23.194:41315): tablet 1699986d9a004834afdcb721bf31beda (table test-table [id=19ddbc5a0eb4424da9255dbbffb51550]) successfully deleted
I20260928 14:36:48.586289 17006 ts_tablet_manager.cc:1918] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:36:48.586409 17006 log.cc:1196] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/1699986d9a004834afdcb721bf31beda
W20260928 14:36:48.586555 16493 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:33027 (error 108)
I20260928 14:36:48.586753 17006 ts_tablet_manager.cc:1939] T 1699986d9a004834afdcb721bf31beda P 6608d162b49e44cda83e4d50714a834c: Deleting consensus metadata
I20260928 14:36:48.586804 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 16703
I20260928 14:36:48.588250 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:36:48.587973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:33027 (local address 127.0.0.1:51556)
0928 14:36:48.588106 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:36:48.588175 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:33027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 14:36:48.588420 16493 catalog_manager.cc:4584] TS 20bb78871cb94fd7b54c5a357bdd95f4 (127.6.23.193:33027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1699986d9a004834afdcb721bf31beda: Network error: Client connection negotiation failed: client connection to 127.6.23.193:33027: connect: Connection refused (error 111)
I20260928 14:36:48.589324 16493 catalog_manager.cc:4841] TS 6608d162b49e44cda83e4d50714a834c (127.6.23.195:36243): tablet 1699986d9a004834afdcb721bf31beda (table test-table [id=19ddbc5a0eb4424da9255dbbffb51550]) successfully deleted
I20260928 14:36:48.593220 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 16837
I20260928 14:36:48.599054 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 16475
2026-09-28T14:36:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2459 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:36:48 dist-test-slave-06lh krb5kdc[17012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:36:48 dist-test-slave-06lh krb5kdc[17012](info): set up 2 sockets
Sep 28 14:36:48 dist-test-slave-06lh krb5kdc[17012](info): commencing operation
krb5kdc: starting...
W20260928 14:36:49.648602 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:36:49 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:36:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.23.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.23.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/rangeradmin_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/rangeradmin_127.6.23.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.23.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.23.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/rangerlookup_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/rangerlookup_127.6.23.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.23.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.23.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/HTTP_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/HTTP_127.6.23.212@KRBTEST.COM.keytab.
I20260928 14:36:49.822834 6239 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpOhn9p/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/postgres -l logfile start
2026-09-28 14:36:52.422 UTC [17042] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:36:52.422 UTC [17042] LOG: listening on IPv4 address "127.6.23.212", port 42273
2026-09-28 14:36:52.425 UTC [17042] LOG: listening on Unix socket "/tmp/.s.PGSQL.42273"
2026-09-28 14:36:52.447 UTC [17045] LOG: database system was shut down at 2026-09-28 14:36:50 UTC
2026-09-28 14:36:52.451 UTC [17042] LOG: database system is ready to accept connections
I20260928 14:36:53.404335 6239 mini_postgres.cc:96] Postgres bound to 42273
2026-09-28 14:36:53.410 UTC [17053] FATAL: database "slave" does not exist
127.6.23.212:42273 - accepting connections
I20260928 14:36:53.411450 6239 mini_ranger.cc:161] Starting Ranger...
I20260928 14:36:53.427134 6239 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:36:53.918181 6239 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:36:53.918275 6239 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-admin
2026-09-28 14:36:54,188 [I] DB FLAVOR :POSTGRES
2026-09-28 14:36:54,189 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:36:54,189 [I] Checking connection..
2026-09-28 14:36:54,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:36:54,457 [I] Checking connection passed.
2026-09-28 14:36:54,457 [I] --------- Verifying version history table ---------
2026-09-28 14:36:54,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:36:54,706 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:36:54,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:36:54,941 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:36:54,941 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:36:54,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:36:55.202 UTC [17117] WARNING: there is no transaction in progress
2026-09-28 14:36:55,208 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:36:55,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:36:55,470 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:36:55,470 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:36:55,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:36:55,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:36:55,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-06lh.c.gcp-upstream.internal','N') ;"
2026-09-28 14:36:56,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:36:56,503 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:36:56,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:36:56,750 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:36:56,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:36:57,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:36:57,253 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:36:57,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:36:58.280 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.297 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.314 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.330 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.349 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.368 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.427 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.434 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.449 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.459 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.470 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.484 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.495 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.501 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.509 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:58.516 UTC [17225] WARNING: there is no transaction in progress
2026-09-28 14:36:59,217 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:36:59,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:36:59,444 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:36:59,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:36:59,685 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:36:59,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:36:59,954 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:36:59,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:37:00,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:37:00,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-06lh.c.gcp-upstream.internal';"
2026-09-28 14:37:00,689 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:37:00,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:37:00,955 [I] DB_PATCHES have already been applied
I20260928 14:37:00.961717 6239 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpOhn9p/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:37:00.961822 6239 mini_ranger.cc:193] Using host: 127.6.23.212
I20260928 14:37:00.965386 6239 mini_ranger.cc:227] Ranger admin URL: http://127.6.23.212:46305
Sep 28, 2026 2:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:37:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:37:02.077421 6239 mini_ranger.cc:160] Time spent starting Ranger: real 8.666s user 0.008s sys 0.003s
Sep 28, 2026 2:37:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:37:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:37:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:37:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:37:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:37:24.655138 6239 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.23.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.23.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/rangerkms_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/rangerkms_127.6.23.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.23.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.23.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.23.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/HTTP_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.23.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/HTTP_127.6.23.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:37:24 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606244, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:37:24.762656 6239 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:37:24.773366 6239 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:37:25.639945 6239 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:37:25.640040 6239 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms
I20260928 14:37:25.640151 6239 mini_ranger_kms.cc:227] Using postgres at 127.6.23.212:42273
2026-09-28 14:37:25,832 [I] DB FLAVOR :POSTGRES
2026-09-28 14:37:25,832 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:37:25,832 [I] Checking connection
2026-09-28 14:37:25,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:37:26,085 [I] connection success
2026-09-28 14:37:26,086 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:37:26,086 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:37:26,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:37:26,323 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:37:26,323 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:37:26,323 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:37:26,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:42273/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:37:26,596 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:37:27.072749 6239 mini_ranger_kms.cc:310] Created kms service
I20260928 14:37:27.283252 6239 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:37:27.327499 6239 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:37:27 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606244, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:37:27.748471 6239 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:37:27.748620 6239 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:37:27.748642 6239 mini_ranger_kms.cc:240] Using host: 127.6.23.212
I20260928 14:37:27.752298 6239 mini_ranger_kms.cc:276] Ranger KMS PID: 17405
I20260928 14:37:27.752432 6239 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.23.212:33481
Sep 28, 2026 2:37:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:37:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:37:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:37:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:37:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:37:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:37:30.037739 6239 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.275s user 0.006s sys 0.169s
I20260928 14:37:30.037897 6239 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:37:30.038036 6239 mini_ranger_kms.cc:401] 127.6.23.212:33481/kms/v1/keys
Sep 28 14:37:30 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.23.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:37:30 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.23.212@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
Sep 28 14:37:30 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.23.212@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
Sep 28 14:37:33 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:37:33.219491 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:44407
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:44407
--ranger_config_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:33481/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:37:33.335671 17456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:37:33.335969 17456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:37:33.336010 17456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:37:33.340188 17456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:37:33.340260 17456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:37:33.340279 17456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:37:33.340299 17456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:37:33.340314 17456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:37:33.346140 17456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.23.212:33481/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:44407
--ranger_config_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:44407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:37:33.347397 17456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:37:33.348500 17456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:37:33.357952 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:37:33.357942 17461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:37:33.358090 17456 server_base.cc:1034] running on GCE node
W20260928 14:37:33.358357 17464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:37:33.358964 17456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:37:33.360231 17456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:37:33.361433 17456 hybrid_clock.cc:648] HybridClock initialized: now 1790606253361410 us; error 44 us; skew 500 ppm
Sep 28 14:37:33 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606253, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:37:33.368716 17456 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:37:33.369737 17456 webserver.cc:460] Webserver started at http://127.6.23.254:40063/ using document root <none> and password file <none>
I20260928 14:37:33.370352 17456 fs_manager.cc:359] Metadata directory not provided
I20260928 14:37:33.370440 17456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:37:33.370661 17456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:37:33 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606253, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:37:33.432646 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "f4276600f69d491aa118bf2d7c5c118a"
format_stamp: "Formatted at 2026-09-28 14:37:33 on dist-test-slave-06lh"
server_key: "d9fd847296628120e2d235fca6cf9749"
server_key_iv: "569ab3c19babd8c7e1ac0674cd919374"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:33.433393 17456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "f4276600f69d491aa118bf2d7c5c118a"
format_stamp: "Formatted at 2026-09-28 14:37:33 on dist-test-slave-06lh"
server_key: "d9fd847296628120e2d235fca6cf9749"
server_key_iv: "569ab3c19babd8c7e1ac0674cd919374"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:33.438318 17456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:37:33.453011 17471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:33.454522 17456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:37:33.454674 17456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "f4276600f69d491aa118bf2d7c5c118a"
format_stamp: "Formatted at 2026-09-28 14:37:33 on dist-test-slave-06lh"
server_key: "d9fd847296628120e2d235fca6cf9749"
server_key_iv: "569ab3c19babd8c7e1ac0674cd919374"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:33.454824 17456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:37:33.480582 17456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:37:33.483688 17456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:37:33.483909 17456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:37:33.492007 17456 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:44407
I20260928 14:37:33.492115 17523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:44407 every 8 connection(s)
I20260928 14:37:33.493109 17456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:37:33.495977 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:37:33.497885 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 17456
I20260928 14:37:33.498011 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:37:33.498302 6239 external_mini_cluster.cc:1428] Setting key f3d7ae58bc48ab0ac8f81fd68ce5bd63
I20260928 14:37:33.503031 17524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a: Bootstrap starting.
I20260928 14:37:33.505623 17524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a: Neither blocks nor log segments found. Creating new log.
I20260928 14:37:33.506518 17524 log.cc:824] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:37:33 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:37:33.508901 17524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a: No bootstrap required, opened a new log
I20260928 14:37:33.512475 17524 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4276600f69d491aa118bf2d7c5c118a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 44407 } }
I20260928 14:37:33.512799 17524 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:37:33.512905 17524 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4276600f69d491aa118bf2d7c5c118a, State: Initialized, Role: FOLLOWER
I20260928 14:37:33.513485 17524 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4276600f69d491aa118bf2d7c5c118a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 44407 } }
I20260928 14:37:33.513661 17524 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:37:33.513751 17524 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:37:33.513880 17524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:37:33.515125 17524 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4276600f69d491aa118bf2d7c5c118a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 44407 } }
I20260928 14:37:33.515527 17524 leader_election.cc:302] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4276600f69d491aa118bf2d7c5c118a; no voters:
I20260928 14:37:33.515889 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:33.499784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.6.23.254:44407)
0928 14:37:33.500241 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:33.500260 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:33.500304 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:37:33.501077 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:33.501101 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:33.501544 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:33.501934 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:37:33.502853 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:33.504336 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:33.505615 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:33.506299 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:33.506462 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:33.509601 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:37:33.509608 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:37:33.509627 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:37:33.509659 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:37:33.512459 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:33.512987 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:33.512994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:33.513000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:33.513135 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:33.513500 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:33.513503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:33.513505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:33.514046 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:37:33.514260 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:33.514565 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 14:37:33.514868 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":127,"threads_started":1}
I20260928 14:37:33.515915 17524 leader_election.cc:288] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:37:33.515983 17529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:37:33.516544 17529 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [term 1 LEADER]: Becoming Leader. State: Replica: f4276600f69d491aa118bf2d7c5c118a, State: Running, Role: LEADER
I20260928 14:37:33.516894 17529 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4276600f69d491aa118bf2d7c5c118a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 44407 } }
I20260928 14:37:33.517566 17524 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:37:33.518201 17529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4276600f69d491aa118bf2d7c5c118a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4276600f69d491aa118bf2d7c5c118a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 44407 } } }
I20260928 14:37:33.518426 17529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [sys.catalog]: This master's current role is: LEADER
I20260928 14:37:33.518918 17529 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4276600f69d491aa118bf2d7c5c118a. Latest consensus state: current_term: 1 leader_uuid: "f4276600f69d491aa118bf2d7c5c118a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4276600f69d491aa118bf2d7c5c118a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 44407 } } }
I20260928 14:37:33.519302 17529 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a [sys.catalog]: This master's current role is: LEADER
I20260928 14:37:33.519640 17538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:37:33.519925 17524 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:37:33.526016 17538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:37:33.535943 17538 catalog_manager.cc:1347] Generated new cluster ID: 2c0f0cc78a0e4799bd88a075c5296fbb
I20260928 14:37:33.536020 17538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:37:33.554236 17538 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:37:33.555716 17538 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:37:33.561669 17538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a: Generated new TSK 0
I20260928 14:37:33.562568 17538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:37:34.223066 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:33481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:44407
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:37:34.364084 17569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:37:34.364486 17569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:37:34.364553 17569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:37:34.370785 17569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:37:34.370877 17569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:37:34.370968 17569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:37:34.378283 17569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.23.212:33481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:44407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:37:34.379525 17569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:37:34.380774 17569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:37:34.393043 17576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:37:34.393043 17577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:37:34.393668 17579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:37:34.394181 17569 server_base.cc:1034] running on GCE node
I20260928 14:37:34.394685 17569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:37:34.395550 17569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:37:34.396754 17569 hybrid_clock.cc:648] HybridClock initialized: now 1790606254396722 us; error 42 us; skew 500 ppm
Sep 28 14:37:34 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:37:34.400199 17569 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:37:34.401329 17569 webserver.cc:460] Webserver started at http://127.6.23.193:38521/ using document root <none> and password file <none>
I20260928 14:37:34.402058 17569 fs_manager.cc:359] Metadata directory not provided
I20260928 14:37:34.402150 17569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:37:34.402398 17569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:37:34 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:37:34.418962 17569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "2b69cf2452c44730b37253199dfee73a"
format_stamp: "Formatted at 2026-09-28 14:37:34 on dist-test-slave-06lh"
server_key: "eaa70b44ad112d69e25668e97a31c97d"
server_key_iv: "34c4ecb633626f55dfc7c64972eeb7bf"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:34.419950 17569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "2b69cf2452c44730b37253199dfee73a"
format_stamp: "Formatted at 2026-09-28 14:37:34 on dist-test-slave-06lh"
server_key: "eaa70b44ad112d69e25668e97a31c97d"
server_key_iv: "34c4ecb633626f55dfc7c64972eeb7bf"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:34.424340 17569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:37:34.445540 17586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:34.447530 17569 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260928 14:37:34.447687 17569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "2b69cf2452c44730b37253199dfee73a"
format_stamp: "Formatted at 2026-09-28 14:37:34 on dist-test-slave-06lh"
server_key: "eaa70b44ad112d69e25668e97a31c97d"
server_key_iv: "34c4ecb633626f55dfc7c64972eeb7bf"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:34.447852 17569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:37:34.483153 17569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:37:34.487905 17569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:37:34.488174 17569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:37:34.488982 17569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:37:34.490146 17569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:37:34.490262 17569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:34.490388 17569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:37:34.490473 17569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:34.501284 17569 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:39647
I20260928 14:37:34.502571 17569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:37:34.509770 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 17569
I20260928 14:37:34.509898 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:37:34.510205 6239 external_mini_cluster.cc:1428] Setting key c08d216e873b0743c87c42c3501be357
I20260928 14:37:34.516121 17702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:39647 every 8 connection(s)
Sep 28 14:37:34 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:37:34.550603 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:34.523920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:44407 (local address 127.6.23.193:52671)
0928 14:37:34.524575 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:34.524646 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 14:37:34.525762 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:37:34.527133 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:37:34.527144 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:37:34.527573 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:37:34.528282 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:34.528302 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:34.533008 (+ 4706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:34.533013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:34.535036 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:34.535055 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:34.536018 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:34.536024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:34.536246 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:34.537064 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:37:34.537089 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:37:34.538999 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:37:34.541411 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:37:34.541418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:37:34.541439 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:37:34.541826 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:37:34.547479 (+ 5653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:37:34.547483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:37:34.547485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:37:34.547763 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:37:34.548385 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:37:34.548393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:37:34.548788 (+ 395us) client_negotiation.cc:769] Sending connection context
0928 14:37:34.549043 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 14:37:34.549458 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":152,"threads_started":1}
I20260928 14:37:34.551411 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:34.523391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:52671 (local address 127.6.23.254:44407)
0928 14:37:34.526623 (+ 3232us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:34.526633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:34.526651 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:37:34.526700 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:34.526703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:34.526756 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:34.528450 (+ 1694us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:37:34.528578 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:34.529436 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:34.535274 (+ 5838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:34.535654 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:34.535777 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:34.539264 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:37:34.539270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:37:34.539275 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:37:34.539310 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:37:34.541256 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:34.547123 (+ 5867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:34.547131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:34.547133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:34.547313 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:34.547904 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:34.547910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:34.547912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:34.548205 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:37:34.548305 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:34.551076 (+ 2771us) server_negotiation.cc:299] Negotiation successful
0928 14:37:34.551221 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3137,"thread_start_us":153,"threads_started":1}
I20260928 14:37:34.553292 17703 heartbeater.cc:347] Connected to a master server at 127.6.23.254:44407
I20260928 14:37:34.553591 17703 heartbeater.cc:464] Registering TS with master...
I20260928 14:37:34.554216 17703 heartbeater.cc:511] Master 127.6.23.254:44407 requested a full tablet report, sending...
I20260928 14:37:34.556092 17488 ts_manager.cc:194] Registered new tserver with Master: 2b69cf2452c44730b37253199dfee73a (127.6.23.193:39647)
I20260928 14:37:34.557644 17488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:52671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:37:34.582697 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:33481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:44407
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:37:34 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 14:37:34.752149 17709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:37:34.752568 17709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:37:34.752635 17709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:37:34.759332 17709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:37:34.759472 17709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:37:34.759583 17709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:37:34.767809 17709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.23.212:33481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:44407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:37:34.769568 17709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:37:34.770987 17709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:37:34.784039 17719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:37:34.784024 17717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:37:34.784802 17716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:37:34.786777 17709 server_base.cc:1034] running on GCE node
I20260928 14:37:34.787478 17709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:37:34.788348 17709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:37:34.789639 17709 hybrid_clock.cc:648] HybridClock initialized: now 1790606254789488 us; error 165 us; skew 500 ppm
Sep 28 14:37:34 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:37:34.793756 17709 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:37:34.794957 17709 webserver.cc:460] Webserver started at http://127.6.23.194:42471/ using document root <none> and password file <none>
I20260928 14:37:34.795764 17709 fs_manager.cc:359] Metadata directory not provided
I20260928 14:37:34.795863 17709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:37:34.796204 17709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:37:34 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:37:34.814683 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d"
format_stamp: "Formatted at 2026-09-28 14:37:34 on dist-test-slave-06lh"
server_key: "97ba4932ae93c4d8b2fcc4a4592c4616"
server_key_iv: "7e77a0a1a3b5c9dab744ec661eb409ae"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:34.815472 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d"
format_stamp: "Formatted at 2026-09-28 14:37:34 on dist-test-slave-06lh"
server_key: "97ba4932ae93c4d8b2fcc4a4592c4616"
server_key_iv: "7e77a0a1a3b5c9dab744ec661eb409ae"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:34.819984 17709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:37:34.838640 17726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:34.841166 17709 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 14:37:34.841348 17709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d"
format_stamp: "Formatted at 2026-09-28 14:37:34 on dist-test-slave-06lh"
server_key: "97ba4932ae93c4d8b2fcc4a4592c4616"
server_key_iv: "7e77a0a1a3b5c9dab744ec661eb409ae"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:34.841533 17709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:37:34.875922 17709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:37:34.879869 17709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:37:34.880127 17709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:37:34.880900 17709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:37:34.882026 17709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:37:34.882143 17709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:34.882252 17709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:37:34.882330 17709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:34.893205 17709 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:43733
I20260928 14:37:34.894479 17709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:37:34.901690 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 17709
I20260928 14:37:34.901831 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:37:34.902138 6239 external_mini_cluster.cc:1428] Setting key bd90631884b9eef298d6ee8e73066c3c
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:37:34.914359 17839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:43733 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Sep 28 14:37:34 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:37:34.939396 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:34.919935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:44407 (local address 127.6.23.194:56531)
0928 14:37:34.921279 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:34.921346 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 14:37:34.922457 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:37:34.923106 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:37:34.923115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:37:34.923547 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:37:34.924250 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:34.924268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:34.925411 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:34.925415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:34.926866 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:34.926888 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:34.930522 (+ 3634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:34.930527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:34.930764 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:34.931514 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:37:34.931538 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:37:34.933433 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:37:34.935813 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:37:34.935819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:37:34.935840 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:37:34.936224 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:37:34.936578 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:37:34.936581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:37:34.936583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:37:34.936777 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:37:34.937290 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:37:34.937297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:37:34.937650 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 14:37:34.937858 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 14:37:34.938241 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":148,"threads_started":1}
I20260928 14:37:34.940171 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:34.919186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:56531 (local address 127.6.23.254:44407)
0928 14:37:34.919339 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:34.919344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:34.921468 (+ 2124us) server_negotiation.cc:407] Connection header received
0928 14:37:34.922704 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:34.922709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:34.922768 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:34.922876 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:37:34.924405 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:34.925254 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:34.927012 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:34.927419 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:34.927528 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:34.933837 (+ 6309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:37:34.933842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:37:34.933844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:37:34.933876 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:37:34.935693 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:34.936360 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:34.936364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:34.936366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:34.936466 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:34.936901 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:34.936904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:34.936907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:34.937141 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:37:34.937235 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:34.939844 (+ 2609us) server_negotiation.cc:299] Negotiation successful
0928 14:37:34.940008 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:37:34.941866 17840 heartbeater.cc:347] Connected to a master server at 127.6.23.254:44407
I20260928 14:37:34.942142 17840 heartbeater.cc:464] Registering TS with master...
I20260928 14:37:34.942967 17840 heartbeater.cc:511] Master 127.6.23.254:44407 requested a full tablet report, sending...
I20260928 14:37:34.945485 17488 ts_manager.cc:194] Registered new tserver with Master: b59781b3df7b40e4b73f4c5ed0e6d93d (127.6.23.194:43733)
I20260928 14:37:34.946395 17488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:56531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:37:34.961810 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:33481/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:44407
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:36485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:37:35 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
W20260928 14:37:35.132287 17845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:37:35.132779 17845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:37:35.132884 17845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:37:35.138543 17845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:37:35.138707 17845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:37:35.138831 17845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:37:35.146427 17845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.23.212:33481/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:44407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:37:35.148053 17845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:37:35.149646 17845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:37:35.162648 17853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:37:35.162846 17851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:37:35.163003 17850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:37:35.163336 17845 server_base.cc:1034] running on GCE node
I20260928 14:37:35.164291 17845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:37:35.165190 17845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:37:35.166782 17845 hybrid_clock.cc:648] HybridClock initialized: now 1790606255166754 us; error 40 us; skew 500 ppm
Sep 28 14:37:35 dist-test-slave-06lh krb5kdc[17012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:37:35.171073 17845 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:37:35.172353 17845 webserver.cc:460] Webserver started at http://127.6.23.195:34069/ using document root <none> and password file <none>
I20260928 14:37:35.173137 17845 fs_manager.cc:359] Metadata directory not provided
I20260928 14:37:35.173393 17845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:37:35.173693 17845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:37:35 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:37:35.197161 17845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "a435d201a01e4ca4a99c5be684922261"
format_stamp: "Formatted at 2026-09-28 14:37:35 on dist-test-slave-06lh"
server_key: "95c65892b362bd7736cb93906dd1ffd4"
server_key_iv: "cb5236e91fbe42983ffc194af79abc83"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:35.197950 17845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "a435d201a01e4ca4a99c5be684922261"
format_stamp: "Formatted at 2026-09-28 14:37:35 on dist-test-slave-06lh"
server_key: "95c65892b362bd7736cb93906dd1ffd4"
server_key_iv: "cb5236e91fbe42983ffc194af79abc83"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:35.217085 17845 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.002s sys 0.004s
I20260928 14:37:35.243705 17860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:35.247579 17845 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260928 14:37:35.247753 17845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "a435d201a01e4ca4a99c5be684922261"
format_stamp: "Formatted at 2026-09-28 14:37:35 on dist-test-slave-06lh"
server_key: "95c65892b362bd7736cb93906dd1ffd4"
server_key_iv: "cb5236e91fbe42983ffc194af79abc83"
server_key_version: "kuduclusterkey@0"
I20260928 14:37:35.247921 17845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:37:35.271646 17845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:37:35.275398 17845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:37:35.275679 17845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:37:35.276453 17845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:37:35.277626 17845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:37:35.277740 17845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:35.277938 17845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:37:35.278020 17845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:37:35.289585 17845 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:44289
I20260928 14:37:35.290930 17845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:37:35.301019 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 17845
I20260928 14:37:35.301168 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:37:35.301496 6239 external_mini_cluster.cc:1428] Setting key bfec72b89948975d1ce1b9ba47fbd5fe
I20260928 14:37:35.308180 17973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:44289 every 8 connection(s)
Sep 28 14:37:35 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:37:35.334884 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.300250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:56643 (local address 127.6.23.254:44407)
0928 14:37:35.300375 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:35.300380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:35.307817 (+ 7437us) server_negotiation.cc:407] Connection header received
0928 14:37:35.311802 (+ 3985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:35.311807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:35.311870 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:35.311924 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:37:35.323402 (+ 11478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.324320 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.326609 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.326968 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.327068 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.330370 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:37:35.330375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:37:35.330377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:37:35.330410 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:37:35.332294 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:35.332995 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:35.332999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:35.333001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:35.333105 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:35.333531 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:35.333534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:35.333536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:35.333767 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:37:35.333846 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:35.334572 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 14:37:35.334713 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:37:35.336932 17976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.299067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:44407 (local address 127.6.23.195:56643)
0928 14:37:35.307592 (+ 8525us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:35.307683 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 14:37:35.311480 (+ 3797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:37:35.312430 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:37:35.312441 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:37:35.312850 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:37:35.313639 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.313662 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.324564 (+ 10902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.324568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.326440 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.326458 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.327351 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.327355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.327595 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.328334 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:37:35.328357 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:37:35.330131 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:37:35.332456 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:37:35.332463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:37:35.332483 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:37:35.332866 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:37:35.333218 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:37:35.333221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:37:35.333223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:37:35.333418 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:37:35.333912 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:37:35.333919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:37:35.334276 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 14:37:35.335164 (+ 888us) client_negotiation.cc:240] Negotiation successful
0928 14:37:35.335621 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8204,"thread_start_us":180,"threads_started":1}
I20260928 14:37:35.338421 17974 heartbeater.cc:347] Connected to a master server at 127.6.23.254:44407
I20260928 14:37:35.338707 17974 heartbeater.cc:464] Registering TS with master...
I20260928 14:37:35.339345 17974 heartbeater.cc:511] Master 127.6.23.254:44407 requested a full tablet report, sending...
I20260928 14:37:35.340528 17488 ts_manager.cc:194] Registered new tserver with Master: a435d201a01e4ca4a99c5be684922261 (127.6.23.195:44289)
I20260928 14:37:35.341280 17488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:56643
I20260928 14:37:35.342499 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:37:35.377259 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.351848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.6.23.254:44407)
0928 14:37:35.352011 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:35.352016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:35.359512 (+ 7496us) server_negotiation.cc:407] Connection header received
0928 14:37:35.359745 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:35.359749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:35.359805 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:35.359858 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:37:35.363364 (+ 3506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.364216 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.365580 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.365945 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.366032 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.372609 (+ 6577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:37:35.372615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:37:35.372618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:37:35.372659 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:37:35.374514 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:35.375100 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:35.375104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:35.375106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:35.375203 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:37:35.376201 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:37:35.376205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:37:35.376207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:37:35.376449 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:37:35.376569 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:35.376937 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 14:37:35.377083 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:37:35.381428 17488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:37:35.385824 17488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:37:35.445983 17991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.426334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:44289 (local address 127.0.0.1:53034)
0928 14:37:35.432272 (+ 5938us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:35.432340 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 14:37:35.432487 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:37:35.436857 (+ 4370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:37:35.436860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:37:35.436889 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:37:35.437004 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.437010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.438462 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.438466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.439942 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.439952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.445313 (+ 5361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.445317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.445490 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.445536 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:37:35.445659 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:37:35.445797 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5720,"mutex_wait_us":4,"thread_start_us":133,"threads_started":1}
I20260928 14:37:35.446696 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.426475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.6.23.195:44289)
0928 14:37:35.436284 (+ 9809us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:35.436295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:35.436317 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:37:35.436386 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:35.436391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:35.436568 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:35.436712 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:37:35.437138 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.438321 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.442511 (+ 4190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.444956 (+ 2445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.445061 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.445241 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:35.446437 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0928 14:37:35.446554 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9697,"thread_start_us":161,"threads_started":1}
I20260928 14:37:35.449620 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.428688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:39647 (local address 127.0.0.1:39582)
0928 14:37:35.433239 (+ 4551us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:35.433251 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:37:35.433331 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:37:35.434157 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:37:35.434163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:37:35.434186 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:37:35.434363 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.434373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.443580 (+ 9207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.443583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.444986 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.445001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.448997 (+ 3996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.449003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.449159 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.449197 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:37:35.449284 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:37:35.449392 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4495,"thread_start_us":80,"threads_started":1}
I20260928 14:37:35.450599 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.429075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39582 (local address 127.6.23.193:39647)
0928 14:37:35.432796 (+ 3721us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:35.432804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:35.433434 (+ 630us) server_negotiation.cc:407] Connection header received
0928 14:37:35.433527 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:35.433533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:35.433789 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:35.433963 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:37:35.434494 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.435655 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.447356 (+ 11701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.448559 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.448703 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.448914 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:35.450235 (+ 1321us) server_negotiation.cc:299] Negotiation successful
0928 14:37:35.450386 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3602,"thread_start_us":83,"threads_started":1}
I20260928 14:37:35.450150 17908 tablet_service.cc:1467] Processing CreateTablet for tablet e396ff6c47e74deb9294c109657ddc2e (DEFAULT_TABLE table=test-table [id=9fd318f676194933afdbf03835fcda48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:37:35.451406 17908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e396ff6c47e74deb9294c109657ddc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:37:35.453845 17636 tablet_service.cc:1467] Processing CreateTablet for tablet e396ff6c47e74deb9294c109657ddc2e (DEFAULT_TABLE table=test-table [id=9fd318f676194933afdbf03835fcda48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:37:35.454980 17636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e396ff6c47e74deb9294c109657ddc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:37:35.460011 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.428061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:43733 (local address 127.0.0.1:55518)
0928 14:37:35.432957 (+ 4896us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:35.432973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:37:35.433062 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:37:35.443300 (+ 10238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:37:35.443304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:37:35.443329 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:37:35.443438 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.443446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.445158 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.445163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.447115 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.447131 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.459430 (+ 12299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.459436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.459617 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.459651 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:37:35.459754 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:37:35.459853 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4832,"thread_start_us":93,"threads_started":1}
I20260928 14:37:35.466238 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.428220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.6.23.194:43733)
0928 14:37:35.435442 (+ 7222us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:35.435458 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:35.435485 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:37:35.435560 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:35.435566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:35.435786 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:35.435971 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:37:35.443622 (+ 7651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.444770 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.454903 (+ 10133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.458987 (+ 4084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.459111 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.459297 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:35.465909 (+ 6612us) server_negotiation.cc:299] Negotiation successful
0928 14:37:35.466057 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7102,"thread_start_us":92,"threads_started":1}
I20260928 14:37:35.467947 17994 tablet_bootstrap.cc:492] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a: Bootstrap starting.
I20260928 14:37:35.469659 17774 tablet_service.cc:1467] Processing CreateTablet for tablet e396ff6c47e74deb9294c109657ddc2e (DEFAULT_TABLE table=test-table [id=9fd318f676194933afdbf03835fcda48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:37:35.470820 17774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e396ff6c47e74deb9294c109657ddc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:37:35.480377 17994 tablet_bootstrap.cc:654] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a: Neither blocks nor log segments found. Creating new log.
I20260928 14:37:35.481492 17994 log.cc:824] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:37:35.481822 17995 tablet_bootstrap.cc:492] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Bootstrap starting.
I20260928 14:37:35.483579 17996 tablet_bootstrap.cc:492] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Bootstrap starting.
I20260928 14:37:35.485726 17994 tablet_bootstrap.cc:492] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a: No bootstrap required, opened a new log
I20260928 14:37:35.486003 17994 ts_tablet_manager.cc:1397] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260928 14:37:35.486270 17996 tablet_bootstrap.cc:654] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Neither blocks nor log segments found. Creating new log.
I20260928 14:37:35.487102 17996 log.cc:824] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Log is configured to *not* fsync() on all Append() calls
I20260928 14:37:35.488533 17995 tablet_bootstrap.cc:654] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Neither blocks nor log segments found. Creating new log.
I20260928 14:37:35.489779 17995 log.cc:824] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:37:35.491384 17994 raft_consensus.cc:348] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
I20260928 14:37:35.491846 17994 raft_consensus.cc:374] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:37:35.491959 17994 raft_consensus.cc:729] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b69cf2452c44730b37253199dfee73a, State: Initialized, Role: FOLLOWER
I20260928 14:37:35.492579 17994 consensus_queue.cc:260] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
Sep 28 14:37:35 dist-test-slave-06lh krb5kdc[17012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606254, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:37:35.496460 17994 ts_tablet_manager.cc:1428] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260928 14:37:35.498818 17703 heartbeater.cc:503] Master 127.6.23.254:44407 was elected leader, sending a full tablet report...
I20260928 14:37:35.499089 17996 tablet_bootstrap.cc:492] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: No bootstrap required, opened a new log
I20260928 14:37:35.499452 17996 ts_tablet_manager.cc:1397] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260928 14:37:35.503374 17996 raft_consensus.cc:348] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
I20260928 14:37:35.503633 17996 raft_consensus.cc:374] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:37:35.503690 17996 raft_consensus.cc:729] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a435d201a01e4ca4a99c5be684922261, State: Initialized, Role: FOLLOWER
I20260928 14:37:35.504159 17996 consensus_queue.cc:260] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
I20260928 14:37:35.504854 17974 heartbeater.cc:503] Master 127.6.23.254:44407 was elected leader, sending a full tablet report...
I20260928 14:37:35.505225 17996 ts_tablet_manager.cc:1428] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:37:35.510277 17995 tablet_bootstrap.cc:492] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: No bootstrap required, opened a new log
I20260928 14:37:35.510530 17995 ts_tablet_manager.cc:1397] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20260928 14:37:35.514647 17995 raft_consensus.cc:348] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
I20260928 14:37:35.514938 17995 raft_consensus.cc:374] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:37:35.514993 17995 raft_consensus.cc:729] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59781b3df7b40e4b73f4c5ed0e6d93d, State: Initialized, Role: FOLLOWER
I20260928 14:37:35.515511 17995 consensus_queue.cc:260] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
I20260928 14:37:35.516644 17995 ts_tablet_manager.cc:1428] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:37:35.517055 17840 heartbeater.cc:503] Master 127.6.23.254:44407 was elected leader, sending a full tablet report...
W20260928 14:37:35.519523 17704 tablet.cc:2369] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:37:35.551126 17975 tablet.cc:2369] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:37:35.657853 17841 tablet.cc:2369] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:37:35.734992 18000 raft_consensus.cc:482] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:37:35.735281 18000 raft_consensus.cc:504] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
I20260928 14:37:35.736773 18000 leader_election.cc:288] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a435d201a01e4ca4a99c5be684922261 (127.6.23.195:44289), b59781b3df7b40e4b73f4c5ed0e6d93d (127.6.23.194:43733)
I20260928 14:37:35.750056 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.738485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:52493 (local address 127.6.23.194:43733)
0928 14:37:35.738640 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:35.738645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:35.743814 (+ 5169us) server_negotiation.cc:407] Connection header received
0928 14:37:35.744059 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:35.744063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:35.744123 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:35.744176 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:37:35.745390 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.746155 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.748727 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.749348 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.749482 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.749536 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:35.749820 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 14:37:35.749903 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:37:35.750821 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.738081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:43733 (local address 127.6.23.193:52493)
0928 14:37:35.743730 (+ 5649us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:35.743743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:37:35.743921 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:37:35.744425 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:37:35.744428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:37:35.745177 (+ 749us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:37:35.745265 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.745272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.747716 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.747723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.748589 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.748598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.749605 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.749609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.749693 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.749714 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:37:35.750577 (+ 863us) client_negotiation.cc:240] Negotiation successful
0928 14:37:35.750663 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5595,"spinlock_wait_cycles":1652096,"thread_start_us":111,"threads_started":1}
I20260928 14:37:35.752177 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e396ff6c47e74deb9294c109657ddc2e" candidate_uuid: "2b69cf2452c44730b37253199dfee73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" is_pre_election: true
I20260928 14:37:35.752601 17781 raft_consensus.cc:2454] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b69cf2452c44730b37253199dfee73a in term 0.
I20260928 14:37:35.753232 17590 leader_election.cc:302] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b69cf2452c44730b37253199dfee73a, b59781b3df7b40e4b73f4c5ed0e6d93d; no voters:
I20260928 14:37:35.753635 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.737634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:60411 (local address 127.6.23.195:44289)
0928 14:37:35.737763 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:35.737769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:35.743601 (+ 5832us) server_negotiation.cc:407] Connection header received
0928 14:37:35.744690 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:35.744693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:35.744745 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:35.744792 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:37:35.746305 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.747046 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.751312 (+ 4266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.751766 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.751844 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.751891 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:35.753392 (+ 1501us) server_negotiation.cc:299] Negotiation successful
0928 14:37:35.753488 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:37:35.753821 18000 raft_consensus.cc:2781] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:37:35.753952 18000 raft_consensus.cc:482] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:37:35.754014 18000 raft_consensus.cc:3037] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:37:35.755208 18000 raft_consensus.cc:504] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
I20260928 14:37:35.755663 18000 leader_election.cc:288] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [CANDIDATE]: Term 1 election: Requested vote from peers a435d201a01e4ca4a99c5be684922261 (127.6.23.195:44289), b59781b3df7b40e4b73f4c5ed0e6d93d (127.6.23.194:43733)
I20260928 14:37:35.756374 17781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e396ff6c47e74deb9294c109657ddc2e" candidate_uuid: "2b69cf2452c44730b37253199dfee73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d"
I20260928 14:37:35.756531 17781 raft_consensus.cc:3037] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:37:35.757786 17781 raft_consensus.cc:2454] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b69cf2452c44730b37253199dfee73a in term 1.
I20260928 14:37:35.758183 17590 leader_election.cc:302] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b69cf2452c44730b37253199dfee73a, b59781b3df7b40e4b73f4c5ed0e6d93d; no voters:
I20260928 14:37:35.758426 18000 raft_consensus.cc:2781] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:37:35.758670 18000 raft_consensus.cc:686] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [term 1 LEADER]: Becoming Leader. State: Replica: 2b69cf2452c44730b37253199dfee73a, State: Running, Role: LEADER
I20260928 14:37:35.758739 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.737201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:44289 (local address 127.6.23.193:60411)
0928 14:37:35.743471 (+ 6270us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:35.743491 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:37:35.744554 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:37:35.744934 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:37:35.744936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:37:35.744953 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:37:35.745061 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.745067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.748772 (+ 3705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.748775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.749864 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:37:35.749871 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.753034 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.753038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:37:35.753127 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.753148 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:37:35.758457 (+ 5309us) client_negotiation.cc:240] Negotiation successful
0928 14:37:35.758561 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6170,"thread_start_us":129,"threads_started":1}
I20260928 14:37:35.759099 18000 consensus_queue.cc:237] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } }
I20260928 14:37:35.759697 17928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e396ff6c47e74deb9294c109657ddc2e" candidate_uuid: "2b69cf2452c44730b37253199dfee73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a435d201a01e4ca4a99c5be684922261" is_pre_election: true
I20260928 14:37:35.760010 17928 raft_consensus.cc:2454] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b69cf2452c44730b37253199dfee73a in term 0.
I20260928 14:37:35.760332 17927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e396ff6c47e74deb9294c109657ddc2e" candidate_uuid: "2b69cf2452c44730b37253199dfee73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a435d201a01e4ca4a99c5be684922261"
I20260928 14:37:35.760452 17927 raft_consensus.cc:3037] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:37:35.761700 17927 raft_consensus.cc:2454] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b69cf2452c44730b37253199dfee73a in term 1.
I20260928 14:37:35.762848 17487 catalog_manager.cc:5495] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b69cf2452c44730b37253199dfee73a (127.6.23.193). New cstate: current_term: 1 leader_uuid: "2b69cf2452c44730b37253199dfee73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b69cf2452c44730b37253199dfee73a" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39647 } health_report { overall_health: HEALTHY } } }
I20260928 14:37:35.805415 17992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.801714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.6.23.193:39647)
0928 14:37:35.801856 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:37:35.801860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:37:35.801886 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:37:35.801977 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:37:35.801979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:37:35.802032 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:37:35.802149 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:37:35.802431 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.802960 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.803904 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:37:35.804122 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:37:35.804232 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:37:35.804559 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:37:35.804985 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:37:35.805114 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:37:35.805188 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 14:37:35.805243 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:37:35.811929 17781 raft_consensus.cc:1261] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 1 FOLLOWER]: Refusing update from remote peer 2b69cf2452c44730b37253199dfee73a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:37:35.811954 17927 raft_consensus.cc:1261] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 1 FOLLOWER]: Refusing update from remote peer 2b69cf2452c44730b37253199dfee73a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:37:35.812686 18008 consensus_queue.cc:1035] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b59781b3df7b40e4b73f4c5ed0e6d93d" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:37:35.812897 18000 consensus_queue.cc:1035] T e396ff6c47e74deb9294c109657ddc2e P 2b69cf2452c44730b37253199dfee73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a435d201a01e4ca4a99c5be684922261" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:37:35.822154 18018 mvcc.cc:205] Tried to move back new op lower bound from 7334323223797960704 to 7334323223595134976. Current Snapshot: MvccSnapshot[applied={T|T < 7334323223797960704}]
I20260928 14:37:35.829759 17488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:37:35.829924 17488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:37:35.832489 17488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4276600f69d491aa118bf2d7c5c118a: Sending DeleteTablet for 3 replicas of tablet e396ff6c47e74deb9294c109657ddc2e
I20260928 14:37:35.833330 17636 tablet_service.cc:1514] Processing DeleteTablet for tablet e396ff6c47e74deb9294c109657ddc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:37:35 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:39582
I20260928 14:37:35.833333 17908 tablet_service.cc:1514] Processing DeleteTablet for tablet e396ff6c47e74deb9294c109657ddc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:37:35 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:53034
I20260928 14:37:35.833370 17774 tablet_service.cc:1514] Processing DeleteTablet for tablet e396ff6c47e74deb9294c109657ddc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:37:35 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:55518
I20260928 14:37:35.833912 18027 tablet_replica.cc:330] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: stopping tablet replica
I20260928 14:37:35.834065 18028 tablet_replica.cc:330] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: stopping tablet replica
I20260928 14:37:35.834149 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 17569
I20260928 14:37:35.834360 18027 raft_consensus.cc:2229] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:37:35.834415 18028 raft_consensus.cc:2229] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:37:35.834806 18028 raft_consensus.cc:2258] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:37:35.834806 18027 raft_consensus.cc:2258] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:37:35.836494 18028 ts_tablet_manager.cc:1905] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:37:35.837549 18027 ts_tablet_manager.cc:1905] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:37:35.838732 18028 ts_tablet_manager.cc:1918] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:37:35.838866 18028 log.cc:1196] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/e396ff6c47e74deb9294c109657ddc2e
I20260928 14:37:35.839147 18028 ts_tablet_manager.cc:1939] T e396ff6c47e74deb9294c109657ddc2e P a435d201a01e4ca4a99c5be684922261: Deleting consensus metadata
I20260928 14:37:35.839861 18027 ts_tablet_manager.cc:1918] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:37:35.839938 18027 log.cc:1196] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/e396ff6c47e74deb9294c109657ddc2e
I20260928 14:37:35.840302 18027 ts_tablet_manager.cc:1939] T e396ff6c47e74deb9294c109657ddc2e P b59781b3df7b40e4b73f4c5ed0e6d93d: Deleting consensus metadata
I20260928 14:37:35.840312 17472 catalog_manager.cc:4841] TS a435d201a01e4ca4a99c5be684922261 (127.6.23.195:44289): tablet e396ff6c47e74deb9294c109657ddc2e (table test-table [id=9fd318f676194933afdbf03835fcda48]) successfully deleted
I20260928 14:37:35.841367 17475 catalog_manager.cc:4841] TS b59781b3df7b40e4b73f4c5ed0e6d93d (127.6.23.194:43733): tablet e396ff6c47e74deb9294c109657ddc2e (table test-table [id=9fd318f676194933afdbf03835fcda48]) successfully deleted
W20260928 14:37:35.844647 17475 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:39647 (error 108)
I20260928 14:37:35.844811 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 17709
I20260928 14:37:35.846333 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:37:35.845973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:39647 (local address 127.0.0.1:39600)
0928 14:37:35.846138 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:37:35.846222 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:39647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 14:37:35.846556 17475 catalog_manager.cc:4584] TS 2b69cf2452c44730b37253199dfee73a (127.6.23.193:39647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e396ff6c47e74deb9294c109657ddc2e: Network error: Client connection negotiation failed: client connection to 127.6.23.193:39647: connect: Connection refused (error 111)
I20260928 14:37:35.850725 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 17845
I20260928 14:37:35.856675 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 17456
I20260928 14:37:35.863076 6239 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:37:36.198127 6239 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:37:36.198256 6239 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:37:36.543782 6239 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:37:36.544 UTC [17042] LOG: received smart shutdown request
2026-09-28 14:37:36.547 UTC [17042] LOG: background worker "logical replication launcher" (PID 17051) exited with exit code 1
2026-09-28 14:37:36.548 UTC [17046] LOG: shutting down
2026-09-28 14:37:36.733 UTC [17042] LOG: database system is shut down
2026-09-28T14:37:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48249 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:37:36 dist-test-slave-06lh krb5kdc[18038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:37:36 dist-test-slave-06lh krb5kdc[18038](info): set up 2 sockets
Sep 28 14:37:36 dist-test-slave-06lh krb5kdc[18038](info): commencing operation
krb5kdc: starting...
W20260928 14:37:37.903627 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:37:37 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:37:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.23.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.23.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/rangeradmin_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/rangeradmin_127.6.23.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.23.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.23.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/rangerlookup_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/rangerlookup_127.6.23.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.23.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.23.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/HTTP_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/HTTP_127.6.23.212@KRBTEST.COM.keytab.
I20260928 14:37:38.081735 6239 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskpOhn9p/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/postgres -l logfile start
2026-09-28 14:37:40.781 UTC [18068] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:37:40.781 UTC [18068] LOG: listening on IPv4 address "127.6.23.212", port 60549
2026-09-28 14:37:40.785 UTC [18068] LOG: listening on Unix socket "/tmp/.s.PGSQL.60549"
2026-09-28 14:37:40.806 UTC [18071] LOG: database system was shut down at 2026-09-28 14:37:39 UTC
2026-09-28 14:37:40.809 UTC [18068] LOG: database system is ready to accept connections
I20260928 14:37:41.751449 6239 mini_postgres.cc:96] Postgres bound to 60549
2026-09-28 14:37:41.757 UTC [18079] FATAL: database "slave" does not exist
127.6.23.212:60549 - accepting connections
I20260928 14:37:41.758311 6239 mini_ranger.cc:161] Starting Ranger...
I20260928 14:37:41.773535 6239 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:37:42.256816 6239 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:37:42.256903 6239 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-admin
2026-09-28 14:37:42,534 [I] DB FLAVOR :POSTGRES
2026-09-28 14:37:42,534 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:37:42,534 [I] Checking connection..
2026-09-28 14:37:42,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:37:42,807 [I] Checking connection passed.
2026-09-28 14:37:42,808 [I] --------- Verifying version history table ---------
2026-09-28 14:37:42,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:37:43,066 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:37:43,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:37:43,304 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:37:43,304 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:37:43,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:37:43.566 UTC [18143] WARNING: there is no transaction in progress
2026-09-28 14:37:43,572 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:37:43,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:37:43,818 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:37:43,818 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:37:43,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:37:44,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:37:44,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-06lh.c.gcp-upstream.internal','N') ;"
2026-09-28 14:37:44,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:37:44,796 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:37:44,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:37:45,044 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:37:45,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:37:45,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:37:45,533 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:37:45,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:37:46.553 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.570 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.587 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.603 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.623 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.644 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.705 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.713 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.728 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.739 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.750 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.765 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.775 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.782 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.790 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:46.799 UTC [18251] WARNING: there is no transaction in progress
2026-09-28 14:37:47,480 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:37:47,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:37:47,704 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:37:47,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:37:47,951 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:37:47,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:37:48,200 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:37:48,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:37:48,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:37:48,715 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-06lh.c.gcp-upstream.internal';"
2026-09-28 14:37:48,961 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:37:48,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:37:49,198 [I] DB_PATCHES have already been applied
I20260928 14:37:49.204644 6239 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-admin:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskpOhn9p/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:37:49.204710 6239 mini_ranger.cc:193] Using host: 127.6.23.212
I20260928 14:37:49.208243 6239 mini_ranger.cc:227] Ranger admin URL: http://127.6.23.212:46741
Sep 28, 2026 2:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:37:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:37:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:37:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:37:50.309363 6239 mini_ranger.cc:160] Time spent starting Ranger: real 8.551s user 0.000s sys 0.010s
Sep 28, 2026 2:38:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:38:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:38:10 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:38:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:38:11 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:38:11.736382 6239 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.23.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.23.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/rangerkms_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.23.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/rangerkms_127.6.23.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.23.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.23.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.23.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/HTTP_127.6.23.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.23.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/HTTP_127.6.23.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:38:11 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606291, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:38:11.829119 6239 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:38:11.839358 6239 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:38:12.826910 6239 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:38:12.827004 6239 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms
I20260928 14:38:12.827028 6239 mini_ranger_kms.cc:227] Using postgres at 127.6.23.212:60549
2026-09-28 14:38:13,011 [I] DB FLAVOR :POSTGRES
2026-09-28 14:38:13,012 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:38:13,012 [I] Checking connection
2026-09-28 14:38:13,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:38:13,257 [I] connection success
2026-09-28 14:38:13,257 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:38:13,257 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:38:13,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:38:13,507 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:38:13,508 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:38:13,508 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:38:13,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.23.212:60549/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:38:13,766 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:38:14.223809 6239 mini_ranger_kms.cc:310] Created kms service
I20260928 14:38:14.397233 6239 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:38:14.443635 6239 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:38:14 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606291, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:14.899600 6239 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:38:14.899758 6239 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms:/tmp/dist-test-taskpOhn9p/build/debug/bin/postgresql.jar:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskpOhn9p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:38:14.899780 6239 mini_ranger_kms.cc:240] Using host: 127.6.23.212
I20260928 14:38:14.903669 6239 mini_ranger_kms.cc:276] Ranger KMS PID: 18431
I20260928 14:38:14.903761 6239 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.23.212:38485
Sep 28, 2026 2:38:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:38:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:38:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:38:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:38:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:38:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:38:17.200222 6239 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.371s user 0.015s sys 0.155s
I20260928 14:38:17.200368 6239 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:38:17.200453 6239 mini_ranger_kms.cc:401] 127.6.23.212:38485/kms/v1/keys
Sep 28 14:38:17 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606297, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.23.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:38:17 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606297, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.23.212@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
Sep 28 14:38:17 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606297, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.23.212@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
Sep 28 14:38:20 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:38:20.279589 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42267
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:42267
--ranger_config_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:38485/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:20.404953 18482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:20.405295 18482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:20.405372 18482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:20.409583 18482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:20.409684 18482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:20.409721 18482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:20.409757 18482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:20.409797 18482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:20.415601 18482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.23.212:38485/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:42267
--ranger_config_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:20.416766 18482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:20.417837 18482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:20.427116 18490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:20.427090 18487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:20.427095 18488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:20.427098 18482 server_base.cc:1034] running on GCE node
I20260928 14:38:20.427954 18482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:20.429194 18482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:20.430406 18482 hybrid_clock.cc:648] HybridClock initialized: now 1790606300430388 us; error 32 us; skew 500 ppm
Sep 28 14:38:20 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606300, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:20.433461 18482 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:38:20.434583 18482 webserver.cc:460] Webserver started at http://127.6.23.254:43377/ using document root <none> and password file <none>
I20260928 14:38:20.435173 18482 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:20.435271 18482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:20.435500 18482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:38:20 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606300, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:20.490309 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "f62808a931044d2bb68b799a290f7a2e"
format_stamp: "Formatted at 2026-09-28 14:38:20 on dist-test-slave-06lh"
server_key: "21c6de98045a8038d4c104cba2944085"
server_key_iv: "b71096017d2b9a457ba8b7b56485a164"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:20.491217 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "f62808a931044d2bb68b799a290f7a2e"
format_stamp: "Formatted at 2026-09-28 14:38:20 on dist-test-slave-06lh"
server_key: "21c6de98045a8038d4c104cba2944085"
server_key_iv: "b71096017d2b9a457ba8b7b56485a164"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:20.496821 18482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:38:20.515475 18497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:20.517262 18482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:38:20.517414 18482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "f62808a931044d2bb68b799a290f7a2e"
format_stamp: "Formatted at 2026-09-28 14:38:20 on dist-test-slave-06lh"
server_key: "21c6de98045a8038d4c104cba2944085"
server_key_iv: "b71096017d2b9a457ba8b7b56485a164"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:20.517565 18482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:20.547135 18482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:20.550885 18482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:20.551117 18482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:20.560021 18482 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:42267
I20260928 14:38:20.560073 18549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:42267 every 8 connection(s)
I20260928 14:38:20.561115 18482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:38:20.564868 18550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:20.568557 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 18482
I20260928 14:38:20.568727 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:38:20.569026 6239 external_mini_cluster.cc:1428] Setting key 0becf4b22e70aa12feeb2ee188be6aaf
I20260928 14:38:20.572599 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Bootstrap starting.
I20260928 14:38:20.575691 18550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:20.576642 18550 log.cc:824] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:38:20 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:20.579169 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: No bootstrap required, opened a new log
I20260928 14:38:20.583442 18550 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } }
I20260928 14:38:20.583771 18550 raft_consensus.cc:374] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:20.583859 18550 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62808a931044d2bb68b799a290f7a2e, State: Initialized, Role: FOLLOWER
I20260928 14:38:20.584079 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:20.570598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.6.23.254:42267)
0928 14:38:20.571043 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:20.571058 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:20.571094 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:38:20.571799 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:20.571818 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:20.572188 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:20.572541 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:20.573367 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:20.574365 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:20.575274 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:20.575800 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:20.576360 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:20.578804 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:20.578810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:20.578824 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:20.578849 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:20.581056 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:20.581526 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:20.581531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:20.581536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:20.581639 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:20.581919 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:20.581925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:20.581926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:20.582324 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:20.582526 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:20.582821 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 14:38:20.583152 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":152,"threads_started":1}
I20260928 14:38:20.584393 18550 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } }
I20260928 14:38:20.584549 18550 raft_consensus.cc:388] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:38:20.584648 18550 raft_consensus.cc:482] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:38:20.584748 18550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:20.585922 18550 raft_consensus.cc:504] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } }
I20260928 14:38:20.586316 18550 leader_election.cc:302] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f62808a931044d2bb68b799a290f7a2e; no voters:
I20260928 14:38:20.586714 18550 leader_election.cc:288] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:38:20.586805 18555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:20.587021 18555 raft_consensus.cc:686] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 1 LEADER]: Becoming Leader. State: Replica: f62808a931044d2bb68b799a290f7a2e, State: Running, Role: LEADER
I20260928 14:38:20.587586 18555 consensus_queue.cc:237] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } }
I20260928 14:38:20.588078 18550 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:20.589344 18556 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f62808a931044d2bb68b799a290f7a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } } }
I20260928 14:38:20.589579 18556 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:20.589843 18556 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f62808a931044d2bb68b799a290f7a2e. Latest consensus state: current_term: 1 leader_uuid: "f62808a931044d2bb68b799a290f7a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } } }
I20260928 14:38:20.589974 18556 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:20.590294 18550 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:38:20.593515 18567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:38:20.597939 18567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:38:20.605198 18567 catalog_manager.cc:1347] Generated new cluster ID: c3c750276c3e407880898406f34d2694
I20260928 14:38:20.605265 18567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:38:20.625963 18567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:38:20.627477 18567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:38:20.637400 18567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Generated new TSK 0
I20260928 14:38:20.638412 18567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:38:21.374640 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:38485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42267
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:21.497974 18595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:21.498271 18595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:21.498320 18595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:21.503422 18595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:21.503509 18595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:21.503613 18595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:38:21.512574 18595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.23.212:38485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:21.514154 18595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:21.515712 18595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:21.529062 18603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:21.529090 18605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:21.529246 18602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:21.529477 18595 server_base.cc:1034] running on GCE node
I20260928 14:38:21.530210 18595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:21.530936 18595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:21.532137 18595 hybrid_clock.cc:648] HybridClock initialized: now 1790606301532105 us; error 57 us; skew 500 ppm
Sep 28 14:38:21 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:21.535110 18595 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:38:21.536316 18595 webserver.cc:460] Webserver started at http://127.6.23.193:34433/ using document root <none> and password file <none>
I20260928 14:38:21.537134 18595 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:21.537281 18595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:21.537578 18595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:38:21 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:21.558689 18595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "c88a069b086e446aaa93ecbb02de91c0"
format_stamp: "Formatted at 2026-09-28 14:38:21 on dist-test-slave-06lh"
server_key: "447d5119235691742f783d331fe4ef04"
server_key_iv: "07454e991b725e62db031ac80d3fd9a5"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:21.559749 18595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "c88a069b086e446aaa93ecbb02de91c0"
format_stamp: "Formatted at 2026-09-28 14:38:21 on dist-test-slave-06lh"
server_key: "447d5119235691742f783d331fe4ef04"
server_key_iv: "07454e991b725e62db031ac80d3fd9a5"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:21.565279 18595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:38:21.593814 18612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:21.595781 18595 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260928 14:38:21.595942 18595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "c88a069b086e446aaa93ecbb02de91c0"
format_stamp: "Formatted at 2026-09-28 14:38:21 on dist-test-slave-06lh"
server_key: "447d5119235691742f783d331fe4ef04"
server_key_iv: "07454e991b725e62db031ac80d3fd9a5"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:21.596098 18595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:21.617372 18595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:21.621703 18595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:21.621939 18595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:21.622730 18595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:21.623935 18595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:21.624060 18595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:21.624192 18595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:21.624274 18595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:21.634893 18595 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:39527
I20260928 14:38:21.635809 18725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:39527 every 8 connection(s)
I20260928 14:38:21.636124 18595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:38:21.639864 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 18595
I20260928 14:38:21.640007 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:38:21.640383 6239 external_mini_cluster.cc:1428] Setting key 6e577b33097cbb5e0552171935cec52e
Sep 28 14:38:21 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:21.659899 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:21.638469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:33415 (local address 127.6.23.254:42267)
0928 14:38:21.639483 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:21.639488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:21.639935 (+ 447us) server_negotiation.cc:407] Connection header received
0928 14:38:21.641221 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:21.641227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:21.641297 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:21.641404 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:21.645142 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:21.646020 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:21.647893 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:21.648278 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:21.648401 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:21.651733 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:21.651737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:21.651740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:21.651770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:21.653723 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:21.654491 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:21.654495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:21.654497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:21.654593 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:21.655059 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:21.655062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:21.655064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:21.655340 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:21.655440 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:21.659518 (+ 4078us) server_negotiation.cc:299] Negotiation successful
0928 14:38:21.659689 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":67,"threads_started":1}
I20260928 14:38:21.662004 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:21.639029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42267 (local address 127.6.23.193:33415)
0928 14:38:21.639772 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:21.639830 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:21.640955 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:21.641741 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:21.641750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:21.642126 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:21.644929 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:21.644955 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:21.646200 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:21.646204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:21.647738 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:21.647751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:21.648633 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:21.648636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:21.648852 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:21.649630 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:21.649653 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:21.651513 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:21.653902 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:21.653911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:21.653931 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:21.654360 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:21.654711 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:21.654714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:21.654716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:21.654940 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:21.655509 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:21.655523 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:21.655887 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 14:38:21.660429 (+ 4542us) client_negotiation.cc:240] Negotiation successful
0928 14:38:21.660858 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":153,"threads_started":1}
I20260928 14:38:21.663982 18727 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42267
I20260928 14:38:21.664273 18727 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:21.664968 18727 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
I20260928 14:38:21.667043 18514 ts_manager.cc:194] Registered new tserver with Master: c88a069b086e446aaa93ecbb02de91c0 (127.6.23.193:39527)
I20260928 14:38:21.668790 18514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:33415
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:38:21.694731 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:38485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42267
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:38:21 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 14:38:21.826822 18734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:21.827286 18734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:21.827404 18734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:21.832568 18734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:21.832661 18734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:21.832767 18734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:38:21.838902 18734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.23.212:38485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:21.840142 18734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:21.841276 18734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:21.851380 18741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:21.851478 18740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:21.851415 18743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:21.851930 18734 server_base.cc:1034] running on GCE node
I20260928 14:38:21.852381 18734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:21.853137 18734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:21.854323 18734 hybrid_clock.cc:648] HybridClock initialized: now 1790606301854287 us; error 48 us; skew 500 ppm
Sep 28 14:38:21 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:21.861320 18734 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:38:21.862444 18734 webserver.cc:460] Webserver started at http://127.6.23.194:41645/ using document root <none> and password file <none>
I20260928 14:38:21.863056 18734 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:21.863147 18734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:21.863451 18734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:38:21 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:21.883054 18734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "102d72dd2c864c37a93d1201d820d7f9"
format_stamp: "Formatted at 2026-09-28 14:38:21 on dist-test-slave-06lh"
server_key: "a1c3e85a02bf69e768f0c8ed3c99b88d"
server_key_iv: "0032544fb474909a2fd45ecaa0cd3da8"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:21.883920 18734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "102d72dd2c864c37a93d1201d820d7f9"
format_stamp: "Formatted at 2026-09-28 14:38:21 on dist-test-slave-06lh"
server_key: "a1c3e85a02bf69e768f0c8ed3c99b88d"
server_key_iv: "0032544fb474909a2fd45ecaa0cd3da8"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:21.889846 18734 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260928 14:38:21.919461 18750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:21.921350 18734 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260928 14:38:21.921487 18734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "102d72dd2c864c37a93d1201d820d7f9"
format_stamp: "Formatted at 2026-09-28 14:38:21 on dist-test-slave-06lh"
server_key: "a1c3e85a02bf69e768f0c8ed3c99b88d"
server_key_iv: "0032544fb474909a2fd45ecaa0cd3da8"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:21.921640 18734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:21.973651 18734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:21.978788 18734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:21.979089 18734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:21.979939 18734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:21.981352 18734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:21.981474 18734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:21.981583 18734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:21.981654 18734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:21.997061 18734 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:33109
I20260928 14:38:21.998421 18734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:38:22.006115 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 18734
I20260928 14:38:22.006253 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:38:22.006573 6239 external_mini_cluster.cc:1428] Setting key 8be9c270289543cd42dae2c716b392a7
I20260928 14:38:22.000021 18863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:33109 every 8 connection(s)
Sep 28 14:38:22 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:38:22.020689 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:22.003124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:34181 (local address 127.6.23.254:42267)
0928 14:38:22.003287 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:22.003292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:22.006671 (+ 3379us) server_negotiation.cc:407] Connection header received
0928 14:38:22.008080 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:22.008083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:22.008134 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:22.008187 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:22.010054 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:22.010716 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:22.011997 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:22.012420 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:22.012535 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:22.015903 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:22.015907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:22.015910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:22.015942 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:22.017831 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:22.018558 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:22.018563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:22.018567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:22.018657 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:22.019110 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:22.019114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:22.019118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:22.019412 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:22.019506 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:22.020297 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 14:38:22.020455 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:38:22.022185 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:22.002052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42267 (local address 127.6.23.194:34181)
0928 14:38:22.006345 (+ 4293us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:22.006487 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:22.007817 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:22.008638 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:22.008648 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:22.009062 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:22.009867 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:22.009892 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:22.010864 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:22.010866 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:22.011870 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:22.011880 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:22.012738 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:22.012743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:22.012970 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:22.013731 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:22.013757 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:22.015656 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:22.017982 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:22.017990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:22.018011 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:22.018422 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:22.018772 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:22.018775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:22.018779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:22.018993 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:22.019584 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:22.019592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:22.019984 (+ 392us) client_negotiation.cc:769] Sending connection context
0928 14:38:22.020924 (+ 940us) client_negotiation.cc:240] Negotiation successful
0928 14:38:22.021271 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3873,"thread_start_us":158,"threads_started":1}
I20260928 14:38:22.023511 18864 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:38:22.023792 18864 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:22.024469 18864 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
I20260928 14:38:22.026453 18514 ts_manager.cc:194] Registered new tserver with Master: 102d72dd2c864c37a93d1201d820d7f9 (127.6.23.194:33109)
I20260928 14:38:22.027261 18514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:34181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:38:22.059263 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:38485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42267
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:22.234318 18869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:22.234768 18869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:22.234861 18869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:22.242445 18869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:22.242559 18869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:22.242664 18869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
Sep 28 14:38:22 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:22.252172 18869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.23.212:38485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:22.253691 18869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:22.255158 18869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:22.269747 18875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:22.270257 18869 server_base.cc:1034] running on GCE node
W20260928 14:38:22.270601 18874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:22.271708 18877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:22.272264 18869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:22.273134 18869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:22.276410 18869 hybrid_clock.cc:648] HybridClock initialized: now 1790606302276322 us; error 95 us; skew 500 ppm
Sep 28 14:38:22 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606302, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:22.281292 18869 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:38:22.282486 18869 webserver.cc:460] Webserver started at http://127.6.23.195:35367/ using document root <none> and password file <none>
I20260928 14:38:22.283322 18869 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:22.283440 18869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:22.283710 18869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:38:22 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606302, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:22.299623 18869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "8da0c0df0fa845b6bc7233e019fc5d03"
format_stamp: "Formatted at 2026-09-28 14:38:22 on dist-test-slave-06lh"
server_key: "b4e3492768cdb7912a2e0a9c288a6dd4"
server_key_iv: "04031e4bc036dc1fd68476f61e9f7463"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:22.300523 18869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "8da0c0df0fa845b6bc7233e019fc5d03"
format_stamp: "Formatted at 2026-09-28 14:38:22 on dist-test-slave-06lh"
server_key: "b4e3492768cdb7912a2e0a9c288a6dd4"
server_key_iv: "04031e4bc036dc1fd68476f61e9f7463"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:22.314538 18869 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.006s sys 0.000s
I20260928 14:38:22.351689 18884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:22.362022 18869 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.000s
I20260928 14:38:22.362228 18869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "8da0c0df0fa845b6bc7233e019fc5d03"
format_stamp: "Formatted at 2026-09-28 14:38:22 on dist-test-slave-06lh"
server_key: "b4e3492768cdb7912a2e0a9c288a6dd4"
server_key_iv: "04031e4bc036dc1fd68476f61e9f7463"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:22.362457 18869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:22.426874 18869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:22.447800 18869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:22.448093 18869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:22.448836 18869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:22.449949 18869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:22.454177 18869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 14:38:22.454344 18869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:22.454411 18869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:22.473594 18869 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:43917
I20260928 14:38:22.475656 18869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:38:22.484148 18997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:43917 every 8 connection(s)
I20260928 14:38:22.484750 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 18869
I20260928 14:38:22.484876 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:38:22.485200 6239 external_mini_cluster.cc:1428] Setting key 9ec9630d42e79dbb000420b602a047fe
Sep 28 14:38:22 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606302, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:22.524780 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:22.492255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:40527 (local address 127.6.23.254:42267)
0928 14:38:22.492393 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:22.492398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:22.500200 (+ 7802us) server_negotiation.cc:407] Connection header received
0928 14:38:22.504761 (+ 4561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:22.504765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:22.504830 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:22.504891 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:22.511521 (+ 6630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:22.512368 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:22.514417 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:22.514795 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:22.514889 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:22.520154 (+ 5265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:22.520159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:22.520162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:22.520196 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:22.522047 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:22.522793 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:22.522797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:22.522799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:22.522893 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:22.523349 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:22.523352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:22.523354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:22.523624 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:22.523713 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:22.524479 (+ 766us) server_negotiation.cc:299] Negotiation successful
0928 14:38:22.524616 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:38:22.526624 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:22.493016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42267 (local address 127.6.23.195:40527)
0928 14:38:22.500020 (+ 7004us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:22.500101 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:22.504448 (+ 4347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:22.505394 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:22.505408 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:22.505870 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:22.506615 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:22.506637 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:22.512580 (+ 5943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:22.512584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:22.514232 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:22.514248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:22.515169 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:22.515174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:22.515479 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:22.517960 (+ 2481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:22.517986 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:22.519881 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:22.522219 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:22.522226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:22.522249 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:22.522671 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:22.523003 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:22.523006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:22.523008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:22.523212 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:22.523790 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:22.523798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:22.524166 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 14:38:22.525072 (+ 906us) client_negotiation.cc:240] Negotiation successful
0928 14:38:22.525475 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6720,"thread_start_us":150,"threads_started":1}
I20260928 14:38:22.528266 18998 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42267
I20260928 14:38:22.528568 18998 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:22.529234 18998 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
I20260928 14:38:22.530542 18514 ts_manager.cc:194] Registered new tserver with Master: 8da0c0df0fa845b6bc7233e019fc5d03 (127.6.23.195:43917)
I20260928 14:38:22.531363 18514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:40527
I20260928 14:38:22.535584 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:38:22.536489 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 18595
I20260928 14:38:22.546255 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 18734
I20260928 14:38:22.555203 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 18869
I20260928 14:38:22.569675 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 18482
I20260928 14:38:22.577742 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42267
--webserver_interface=127.6.23.254
--webserver_port=43377
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:42267
--ranger_config_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:38485/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.23.212:38485/kms with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:22.697947 19001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:22.698256 19001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:22.698297 19001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:22.702473 19001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:22.702548 19001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:22.702569 19001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:22.702585 19001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:22.702625 19001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:22.708719 19001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.23.212:38485/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:42267
--ranger_config_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=43377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:22.709892 19001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:22.711016 19001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:22.719616 19007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:22.719722 19006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:22.719734 19009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:22.719978 19001 server_base.cc:1034] running on GCE node
I20260928 14:38:22.720547 19001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:22.721692 19001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:22.722888 19001 hybrid_clock.cc:648] HybridClock initialized: now 1790606302722868 us; error 32 us; skew 500 ppm
Sep 28 14:38:22 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606302, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:22.725883 19001 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:38:22.726866 19001 webserver.cc:460] Webserver started at http://127.6.23.254:43377/ using document root <none> and password file <none>
I20260928 14:38:22.727495 19001 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:22.727568 19001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:38:22 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606302, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:22.792212 19001 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:38:22.795209 19017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:22.796967 19001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:38:22.797122 19001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "f62808a931044d2bb68b799a290f7a2e"
format_stamp: "Formatted at 2026-09-28 14:38:20 on dist-test-slave-06lh"
server_key: "21c6de98045a8038d4c104cba2944085"
server_key_iv: "b71096017d2b9a457ba8b7b56485a164"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:22.797924 19001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:22.819582 19001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:22.823326 19001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:22.823556 19001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:22.832580 19001 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:42267
I20260928 14:38:22.832603 19069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:42267 every 8 connection(s)
I20260928 14:38:22.834064 19001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:38:22.836999 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19001
I20260928 14:38:22.837546 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:38485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:39527
--local_ip_for_outbound_sockets=127.6.23.193
--tserver_master_addrs=127.6.23.254:42267
--webserver_port=34433
--webserver_interface=127.6.23.193
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.23.212:38485/kms with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:38:22.839622 19070 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:38:22.841508 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Bootstrap starting.
I20260928 14:38:22.865253 19070 log.cc:824] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:22.873791 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:38:22.874289 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Bootstrap complete.
I20260928 14:38:22.879849 19070 raft_consensus.cc:348] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } }
I20260928 14:38:22.880360 19070 raft_consensus.cc:729] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62808a931044d2bb68b799a290f7a2e, State: Initialized, Role: FOLLOWER
I20260928 14:38:22.880934 19070 consensus_queue.cc:260] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } }
I20260928 14:38:22.881079 19070 raft_consensus.cc:388] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:38:22.881147 19070 raft_consensus.cc:482] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:38:22.881243 19070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:38:22.883963 19070 raft_consensus.cc:504] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } }
I20260928 14:38:22.884361 19070 leader_election.cc:302] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f62808a931044d2bb68b799a290f7a2e; no voters:
I20260928 14:38:22.884711 19070 leader_election.cc:288] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:38:22.884867 19074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:38:22.885156 19074 raft_consensus.cc:686] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [term 2 LEADER]: Becoming Leader. State: Replica: f62808a931044d2bb68b799a290f7a2e, State: Running, Role: LEADER
I20260928 14:38:22.885501 19074 consensus_queue.cc:237] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } }
I20260928 14:38:22.886034 19070 sys_catalog.cc:564] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:22.887084 19075 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f62808a931044d2bb68b799a290f7a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } } }
I20260928 14:38:22.887084 19076 sys_catalog.cc:455] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f62808a931044d2bb68b799a290f7a2e. Latest consensus state: current_term: 2 leader_uuid: "f62808a931044d2bb68b799a290f7a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f62808a931044d2bb68b799a290f7a2e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42267 } } }
I20260928 14:38:22.887259 19076 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:22.887259 19075 sys_catalog.cc:458] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:22.888535 19083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:38:22.889012 19070 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:38:22.894649 19083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:38:22.895494 19083 catalog_manager.cc:1259] Loaded cluster ID: c3c750276c3e407880898406f34d2694
I20260928 14:38:22.895555 19083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:38:22.898037 19083 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:38:22.898713 19083 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Loaded TSK: 0
I20260928 14:38:22.900028 19083 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:38:22.996491 19072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:22.996871 19072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:22.996949 19072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:23.002962 19072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:23.003074 19072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:23.003190 19072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:38:23.009734 19072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.23.212:38485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:39527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=34433
--tserver_master_addrs=127.6.23.254:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:23.010900 19072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:23.012012 19072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:23.022686 19110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:23.022686 19108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:23.023099 19107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:23.023211 19072 server_base.cc:1034] running on GCE node
I20260928 14:38:23.023903 19072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:23.024694 19072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:23.028055 19072 hybrid_clock.cc:648] HybridClock initialized: now 1790606303028025 us; error 49 us; skew 500 ppm
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:23.031984 19072 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:38:23.033234 19072 webserver.cc:460] Webserver started at http://127.6.23.193:34433/ using document root <none> and password file <none>
I20260928 14:38:23.034018 19072 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:23.034142 19072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:23.068977 19072 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260928 14:38:23.078976 19118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.080971 19072 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260928 14:38:23.081133 19072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "c88a069b086e446aaa93ecbb02de91c0"
format_stamp: "Formatted at 2026-09-28 14:38:21 on dist-test-slave-06lh"
server_key: "447d5119235691742f783d331fe4ef04"
server_key_iv: "07454e991b725e62db031ac80d3fd9a5"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:23.081946 19072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:23.110497 19072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:23.118014 19072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:23.118237 19072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:23.119027 19072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:23.120147 19072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:23.120204 19072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.120260 19072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:23.120281 19072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.133309 19072 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:39527
I20260928 14:38:23.134220 19231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:39527 every 8 connection(s)
I20260928 14:38:23.134732 19072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:38:23.138382 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19072
I20260928 14:38:23.138859 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:38485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:33109
--local_ip_for_outbound_sockets=127.6.23.194
--tserver_master_addrs=127.6.23.254:42267
--webserver_port=41645
--webserver_interface=127.6.23.194
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.23.212:38485/kms with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:23.165333 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:23.137329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42267 (local address 127.6.23.193:40227)
0928 14:38:23.138067 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:23.138136 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:23.139314 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:23.141496 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:23.141510 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:23.141886 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:23.142562 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:23.142586 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.144689 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.144693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:23.146155 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:23.146170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.149203 (+ 3033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.149208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:23.149434 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:23.150433 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:23.150458 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:23.152738 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:23.160663 (+ 7925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:23.160672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:23.160694 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:23.161214 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:23.161799 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:23.161802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:23.161804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:23.162037 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:23.162998 (+ 961us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:23.163007 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:23.163412 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 14:38:23.163632 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 14:38:23.164145 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":131,"threads_started":1}
I20260928 14:38:23.167821 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:23.137164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:40227 (local address 127.6.23.254:42267)
0928 14:38:23.137812 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:23.137836 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:23.139675 (+ 1839us) server_negotiation.cc:407] Connection header received
0928 14:38:23.140028 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:23.140045 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:23.140529 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:23.140915 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:23.142775 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.144445 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.146808 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.147677 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.147871 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:23.153066 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:23.153076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:23.153100 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:23.153141 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:23.155505 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:23.161429 (+ 5924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:23.161437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:23.161446 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:23.161658 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:23.162167 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:23.162171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:23.162173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:23.162768 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:23.162921 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:23.166132 (+ 3211us) server_negotiation.cc:299] Negotiation successful
0928 14:38:23.166529 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":136,"threads_started":1}
I20260928 14:38:23.170854 19232 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42267
I20260928 14:38:23.171161 19232 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:23.171842 19232 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
W20260928 14:38:23.174234 19232 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:42267 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:38:23.174384 19232 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:23.174611 19232 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
I20260928 14:38:23.175570 19232 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:23.175769 19232 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
W20260928 14:38:23.176663 19232 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:38:23.293732 19236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:23.294049 19236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:23.294142 19236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:23.298722 19236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:23.298823 19236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:23.298923 19236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:38:23.305820 19236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.23.212:38485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:33109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=41645
--tserver_master_addrs=127.6.23.254:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:23.307041 19236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:23.308236 19236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:23.318918 19242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:23.318965 19244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:23.319401 19241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:23.320055 19236 server_base.cc:1034] running on GCE node
I20260928 14:38:23.320576 19236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:23.321379 19236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:23.322592 19236 hybrid_clock.cc:648] HybridClock initialized: now 1790606303322556 us; error 53 us; skew 500 ppm
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:23.326562 19236 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:38:23.327817 19236 webserver.cc:460] Webserver started at http://127.6.23.194:41645/ using document root <none> and password file <none>
I20260928 14:38:23.328522 19236 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:23.328649 19236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:23.359218 19236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 14:38:23.362469 19252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.364163 19236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:38:23.364296 19236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "102d72dd2c864c37a93d1201d820d7f9"
format_stamp: "Formatted at 2026-09-28 14:38:21 on dist-test-slave-06lh"
server_key: "a1c3e85a02bf69e768f0c8ed3c99b88d"
server_key_iv: "0032544fb474909a2fd45ecaa0cd3da8"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:23.365027 19236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:23.385149 19236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:23.389271 19236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:23.389484 19236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:23.390225 19236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:23.391352 19236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:23.391414 19236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.391469 19236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:23.391498 19236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.402323 19236 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:33109
I20260928 14:38:23.402482 19365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:33109 every 8 connection(s)
I20260928 14:38:23.403789 19236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:38:23.412223 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19236
I20260928 14:38:23.412752 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.23.212:38485/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:43917
--local_ip_for_outbound_sockets=127.6.23.195
--tserver_master_addrs=127.6.23.254:42267
--webserver_port=35367
--webserver_interface=127.6.23.195
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.23.212:38485/kms with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:23.425648 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:23.407819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:53527 (local address 127.6.23.254:42267)
0928 14:38:23.407977 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:23.407982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:23.411176 (+ 3194us) server_negotiation.cc:407] Connection header received
0928 14:38:23.412526 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:23.412530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:23.412591 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:23.412644 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:23.414400 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.415213 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.416963 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.417319 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.417453 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:23.420878 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:23.420886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:23.420890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:23.420928 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:23.422827 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:23.423594 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:23.423598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:23.423600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:23.423698 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:23.424175 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:23.424178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:23.424180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:23.424437 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:23.424529 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:23.425313 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 14:38:23.425462 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:38:23.427814 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:23.406578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42267 (local address 127.6.23.194:53527)
0928 14:38:23.410969 (+ 4391us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:23.411047 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:23.412246 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:23.413097 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:23.413107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:23.413478 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:23.414209 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:23.414230 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.415397 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.415400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:23.416815 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:23.416826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.417518 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.417524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:23.417751 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:23.418687 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:23.418711 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:23.420624 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:23.422976 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:23.422985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:23.423005 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:23.423456 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:23.423830 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:23.423833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:23.423835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:23.424057 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:23.424599 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:23.424606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:23.424996 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 14:38:23.426184 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0928 14:38:23.426593 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4127,"thread_start_us":154,"threads_started":1}
I20260928 14:38:23.429806 19366 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42267
I20260928 14:38:23.430096 19366 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:23.430739 19366 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
W20260928 14:38:23.432644 19366 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:42267 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:38:23.432787 19366 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:23.433009 19366 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
I20260928 14:38:23.433879 19366 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:23.434087 19366 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
W20260928 14:38:23.434834 19366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:38:23.556741 19370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:23.557154 19370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:23.557225 19370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:23.563529 19370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:23.563609 19370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:23.563699 19370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:38:23.572240 19370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.23.212:38485/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:43917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=35367
--tserver_master_addrs=127.6.23.254:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:23.573725 19370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:23.575110 19370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:38:23.587751 19370 server_base.cc:1034] running on GCE node
W20260928 14:38:23.587759 19376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:23.588155 19375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:23.587836 19378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:23.588754 19370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:23.589438 19370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:23.590627 19370 hybrid_clock.cc:648] HybridClock initialized: now 1790606303590591 us; error 46 us; skew 500 ppm
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:23.594154 19370 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:38:23.595335 19370 webserver.cc:460] Webserver started at http://127.6.23.195:35367/ using document root <none> and password file <none>
I20260928 14:38:23.596053 19370 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:23.596160 19370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:23.621089 19370 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:38:23.624439 19386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.626132 19370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:38:23.626282 19370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "8da0c0df0fa845b6bc7233e019fc5d03"
format_stamp: "Formatted at 2026-09-28 14:38:22 on dist-test-slave-06lh"
server_key: "b4e3492768cdb7912a2e0a9c288a6dd4"
server_key_iv: "04031e4bc036dc1fd68476f61e9f7463"
server_key_version: "kuduclusterkey@0"
I20260928 14:38:23.627022 19370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:23.653241 19370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:23.657991 19370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:23.658250 19370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:23.658893 19370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:23.659910 19370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:23.659981 19370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.660080 19370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:23.660131 19370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:23.671351 19499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:43917 every 8 connection(s)
I20260928 14:38:23.671324 19370 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:43917
I20260928 14:38:23.672897 19370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:38:23.675415 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19370
Sep 28 14:38:23 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:23.692926 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:23.675554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:42267 (local address 127.6.23.195:56215)
0928 14:38:23.677298 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:23.677360 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:23.678498 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:23.679373 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:23.679383 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:23.679767 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:23.680480 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:23.680502 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.681693 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.681696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:23.682753 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:23.682762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.683531 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.683538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:23.683761 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:23.684479 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:23.684507 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:23.686280 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:23.689494 (+ 3214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:23.689502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:23.689519 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:23.689874 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:23.690213 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:23.690217 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:23.690219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:23.690407 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:23.690892 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:23.690899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:23.691219 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 14:38:23.691482 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 14:38:23.691826 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1131,"thread_start_us":155,"threads_started":1}
I20260928 14:38:23.694048 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:23.676925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.6.23.254:42267)
0928 14:38:23.683022 (+ 6097us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:23.683026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:23.683355 (+ 329us) server_negotiation.cc:407] Connection header received
0928 14:38:23.683562 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:23.683565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:23.683612 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:23.683673 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:23.685181 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.685728 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.687649 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.687990 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.688067 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:23.690873 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:23.690877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:23.690880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:23.690905 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:23.692342 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:23.692839 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:23.692843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:23.692844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:23.692930 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:23.693199 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:23.693204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:23.693205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:23.693373 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:23.693581 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:23.693743 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 14:38:23.693862 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6015,"thread_start_us":97,"threads_started":1}
I20260928 14:38:23.694192 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:23.674996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:56215 (local address 127.6.23.254:42267)
0928 14:38:23.675183 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:23.675189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:23.677507 (+ 2318us) server_negotiation.cc:407] Connection header received
0928 14:38:23.678775 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:23.678779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:23.678838 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:23.678893 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:23.680671 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.681524 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.682876 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:23.683195 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:23.683343 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:23.686527 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:23.686531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:23.686534 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:23.686563 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:23.689363 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:23.689997 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:23.690001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:23.690003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:23.690095 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:23.690522 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:23.690525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:23.690527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:23.690736 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:23.690824 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:23.693890 (+ 3066us) server_negotiation.cc:299] Negotiation successful
0928 14:38:23.694007 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 14:38:23.695109 19500 heartbeater.cc:347] Connected to a master server at 127.6.23.254:42267
I20260928 14:38:23.695427 19500 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:23.696015 19500 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
I20260928 14:38:23.697278 19034 ts_manager.cc:194] Registered new tserver with Master: 8da0c0df0fa845b6bc7233e019fc5d03 (127.6.23.195:43917)
I20260928 14:38:23.698422 19034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:56215
Sep 28 14:38:24 dist-test-slave-06lh krb5kdc[18038](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606304, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:24.176898 19232 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:24.177238 19232 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
I20260928 14:38:24.178125 19034 ts_manager.cc:194] Registered new tserver with Master: c88a069b086e446aaa93ecbb02de91c0 (127.6.23.193:39527)
I20260928 14:38:24.178889 19034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:40227
I20260928 14:38:24.435111 19366 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:24.435541 19366 heartbeater.cc:511] Master 127.6.23.254:42267 requested a full tablet report, sending...
I20260928 14:38:24.436405 19033 ts_manager.cc:194] Registered new tserver with Master: 102d72dd2c864c37a93d1201d820d7f9 (127.6.23.194:33109)
I20260928 14:38:24.437229 19033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:53527
I20260928 14:38:24.437358 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:38:24.449826 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.439449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.6.23.254:42267)
0928 14:38:24.440282 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:24.440287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:24.440538 (+ 251us) server_negotiation.cc:407] Connection header received
0928 14:38:24.440737 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:24.440740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:24.440795 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:24.440863 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:24.441872 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.442738 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.444043 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.444396 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.444481 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.446077 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:24.446081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:24.446084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:24.446115 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:24.447745 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:24.448205 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:24.448209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:24.448211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:24.448288 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:24.448647 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:24.448651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:24.448652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:24.448987 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:24.449104 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:24.449490 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 14:38:24.449625 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":98,"threads_started":1}
I20260928 14:38:24.455576 19033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:38:24.459506 19033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 28 14:38:24 dist-test-slave-06lh krb5kdc[18038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606304, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for HTTP/127.6.23.212@KRBTEST.COM
I20260928 14:38:24.478775 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.470937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:33109 (local address 127.0.0.1:52618)
0928 14:38:24.472185 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:24.472239 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:24.472367 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:24.474129 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:24.474138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:24.474161 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:24.474306 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.474314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.475883 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.475886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.477171 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.477181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.478273 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.478277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.478392 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.478434 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:38:24.478517 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:38:24.478624 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":129,"threads_started":1}
I20260928 14:38:24.479884 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.473160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.6.23.194:33109)
0928 14:38:24.473434 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:24.473442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:24.473467 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:38:24.473546 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:24.473551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:24.473775 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:24.473923 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:24.474449 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.475768 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.477315 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.478141 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.479412 (+ 1271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.479566 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:24.479628 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:38:24.479734 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":111,"threads_started":1}
I20260928 14:38:24.481365 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.472298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:39527 (local address 127.0.0.1:55962)
0928 14:38:24.474875 (+ 2577us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:24.474888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:24.474980 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:24.475630 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:24.475633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:24.475648 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:24.475734 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.475739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.477831 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.477834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.478904 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.478917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.480986 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.480989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.481075 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.481099 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:38:24.481168 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:38:24.481238 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2521,"thread_start_us":80,"threads_started":1}
I20260928 14:38:24.481864 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.472840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.6.23.193:39527)
0928 14:38:24.474746 (+ 1906us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:24.474752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:24.475070 (+ 318us) server_negotiation.cc:407] Connection header received
0928 14:38:24.475146 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:24.475150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:24.475359 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:24.475504 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:24.475858 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.477708 (+ 1850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.479037 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.480680 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.480786 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.480924 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:24.481636 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 14:38:24.481734 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1802,"thread_start_us":100,"threads_started":1}
I20260928 14:38:24.483407 19281 tablet_service.cc:1467] Processing CreateTablet for tablet 72e5e896ea334dbdbe0b88f69b994ec5 (DEFAULT_TABLE table=test-table [id=30b65517e6814794a3da48255d7b0252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:24.484661 19281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e5e896ea334dbdbe0b88f69b994ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:24.484829 19166 tablet_service.cc:1467] Processing CreateTablet for tablet 72e5e896ea334dbdbe0b88f69b994ec5 (DEFAULT_TABLE table=test-table [id=30b65517e6814794a3da48255d7b0252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:24.485921 19166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e5e896ea334dbdbe0b88f69b994ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:24.489789 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.471795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:43917 (local address 127.0.0.1:43956)
0928 14:38:24.474405 (+ 2610us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:24.474418 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:24.474515 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:24.483359 (+ 8844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:24.483362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:24.483401 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:24.483516 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.483522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.485908 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.485911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.487150 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.487166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.489357 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.489360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.489484 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.489514 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:38:24.489566 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 14:38:24.489642 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2529,"thread_start_us":108,"threads_started":1}
I20260928 14:38:24.490590 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.472698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.6.23.195:43917)
0928 14:38:24.482852 (+ 10154us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:24.482859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:24.482884 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:38:24.482963 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:24.482969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:24.483179 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:24.484517 (+ 1338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:24.484670 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.485767 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.487415 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.488228 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.489995 (+ 1767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.490164 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:24.490216 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 14:38:24.490317 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10039,"thread_start_us":117,"threads_started":1}
I20260928 14:38:24.493628 19434 tablet_service.cc:1467] Processing CreateTablet for tablet 72e5e896ea334dbdbe0b88f69b994ec5 (DEFAULT_TABLE table=test-table [id=30b65517e6814794a3da48255d7b0252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:24.494748 19434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72e5e896ea334dbdbe0b88f69b994ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:24.496582 19534 tablet_bootstrap.cc:492] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9: Bootstrap starting.
I20260928 14:38:24.499006 19534 tablet_bootstrap.cc:654] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:24.499939 19534 log.cc:824] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:24.502731 19534 tablet_bootstrap.cc:492] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9: No bootstrap required, opened a new log
I20260928 14:38:24.502959 19534 ts_tablet_manager.cc:1397] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260928 14:38:24.503602 19533 tablet_bootstrap.cc:492] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0: Bootstrap starting.
I20260928 14:38:24.506171 19533 tablet_bootstrap.cc:654] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:24.507131 19533 log.cc:824] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:24.507335 19534 raft_consensus.cc:348] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.507637 19534 raft_consensus.cc:374] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:24.507701 19534 raft_consensus.cc:729] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 102d72dd2c864c37a93d1201d820d7f9, State: Initialized, Role: FOLLOWER
I20260928 14:38:24.508181 19534 consensus_queue.cc:260] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.509325 19534 ts_tablet_manager.cc:1428] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:38:24.509738 19366 heartbeater.cc:503] Master 127.6.23.254:42267 was elected leader, sending a full tablet report...
I20260928 14:38:24.515972 19538 tablet_bootstrap.cc:492] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: Bootstrap starting.
I20260928 14:38:24.518726 19533 tablet_bootstrap.cc:492] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0: No bootstrap required, opened a new log
I20260928 14:38:24.518955 19533 ts_tablet_manager.cc:1397] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260928 14:38:24.523589 19533 raft_consensus.cc:348] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.523921 19533 raft_consensus.cc:374] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:24.523972 19533 raft_consensus.cc:729] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c88a069b086e446aaa93ecbb02de91c0, State: Initialized, Role: FOLLOWER
I20260928 14:38:24.524433 19533 consensus_queue.cc:260] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.525068 19538 tablet_bootstrap.cc:654] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:24.525496 19533 ts_tablet_manager.cc:1428] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:38:24.525861 19232 heartbeater.cc:503] Master 127.6.23.254:42267 was elected leader, sending a full tablet report...
I20260928 14:38:24.525955 19538 log.cc:824] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:24.529389 19538 tablet_bootstrap.cc:492] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: No bootstrap required, opened a new log
I20260928 14:38:24.529614 19538 ts_tablet_manager.cc:1397] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260928 14:38:24.534077 19538 raft_consensus.cc:348] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.534395 19538 raft_consensus.cc:374] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:24.534474 19538 raft_consensus.cc:729] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da0c0df0fa845b6bc7233e019fc5d03, State: Initialized, Role: FOLLOWER
I20260928 14:38:24.535029 19538 consensus_queue.cc:260] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.535739 19500 heartbeater.cc:503] Master 127.6.23.254:42267 was elected leader, sending a full tablet report...
I20260928 14:38:24.536175 19538 ts_tablet_manager.cc:1428] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:38:24.597556 19539 raft_consensus.cc:482] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:24.597802 19539 raft_consensus.cc:504] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.599037 19539 leader_election.cc:288] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 102d72dd2c864c37a93d1201d820d7f9 (127.6.23.194:33109), 8da0c0df0fa845b6bc7233e019fc5d03 (127.6.23.195:43917)
I20260928 14:38:24.604235 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.599412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:40295 (local address 127.6.23.194:33109)
0928 14:38:24.600018 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:24.600023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:24.600039 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:38:24.600094 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:24.600098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:24.600151 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:24.600227 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:24.600561 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.601339 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.602909 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.603556 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.603638 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.603694 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:24.604014 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 14:38:24.604091 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
I20260928 14:38:24.604846 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.600068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:48095 (local address 127.6.23.195:43917)
0928 14:38:24.600185 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:24.600192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:24.600570 (+ 378us) server_negotiation.cc:407] Connection header received
0928 14:38:24.600747 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:24.600750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:24.600801 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:24.600848 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:24.601206 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.601927 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.603470 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.604100 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.604199 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.604239 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:24.604641 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 14:38:24.604707 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:38:24.604887 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.599246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:33109 (local address 127.6.23.193:40295)
0928 14:38:24.599655 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:24.599670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:24.599779 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:24.600245 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:24.600248 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:24.600266 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:24.600419 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.600427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.601471 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.601474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.602770 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.602780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.603779 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.603782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.603872 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.603892 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:38:24.604630 (+ 738us) client_negotiation.cc:240] Negotiation successful
0928 14:38:24.604702 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":30,"thread_start_us":136,"threads_started":1}
I20260928 14:38:24.605273 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.599704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:43917 (local address 127.6.23.193:48095)
0928 14:38:24.600492 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:24.600507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:24.600634 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:24.600970 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:24.600973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:24.600990 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:24.601096 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.601101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.602064 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.602067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.603327 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:24.603337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.604391 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.604396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:24.604516 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.604537 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:38:24.605043 (+ 506us) client_negotiation.cc:240] Negotiation successful
0928 14:38:24.605126 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260928 14:38:24.605865 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e5e896ea334dbdbe0b88f69b994ec5" candidate_uuid: "c88a069b086e446aaa93ecbb02de91c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102d72dd2c864c37a93d1201d820d7f9" is_pre_election: true
I20260928 14:38:24.605866 19454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e5e896ea334dbdbe0b88f69b994ec5" candidate_uuid: "c88a069b086e446aaa93ecbb02de91c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" is_pre_election: true
I20260928 14:38:24.606190 19312 raft_consensus.cc:2454] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c88a069b086e446aaa93ecbb02de91c0 in term 0.
I20260928 14:38:24.606190 19454 raft_consensus.cc:2454] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c88a069b086e446aaa93ecbb02de91c0 in term 0.
I20260928 14:38:24.606715 19122 leader_election.cc:302] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8da0c0df0fa845b6bc7233e019fc5d03, c88a069b086e446aaa93ecbb02de91c0; no voters:
I20260928 14:38:24.607065 19539 raft_consensus.cc:2781] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:38:24.607151 19539 raft_consensus.cc:482] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:38:24.607193 19539 raft_consensus.cc:3037] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:24.608390 19539 raft_consensus.cc:504] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.608791 19539 leader_election.cc:288] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [CANDIDATE]: Term 1 election: Requested vote from peers 102d72dd2c864c37a93d1201d820d7f9 (127.6.23.194:33109), 8da0c0df0fa845b6bc7233e019fc5d03 (127.6.23.195:43917)
I20260928 14:38:24.609139 19312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e5e896ea334dbdbe0b88f69b994ec5" candidate_uuid: "c88a069b086e446aaa93ecbb02de91c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "102d72dd2c864c37a93d1201d820d7f9"
I20260928 14:38:24.609202 19454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72e5e896ea334dbdbe0b88f69b994ec5" candidate_uuid: "c88a069b086e446aaa93ecbb02de91c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da0c0df0fa845b6bc7233e019fc5d03"
I20260928 14:38:24.609267 19312 raft_consensus.cc:3037] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:24.609310 19454 raft_consensus.cc:3037] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:24.610551 19312 raft_consensus.cc:2454] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c88a069b086e446aaa93ecbb02de91c0 in term 1.
I20260928 14:38:24.610551 19454 raft_consensus.cc:2454] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c88a069b086e446aaa93ecbb02de91c0 in term 1.
I20260928 14:38:24.610862 19122 leader_election.cc:302] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 102d72dd2c864c37a93d1201d820d7f9, c88a069b086e446aaa93ecbb02de91c0; no voters:
I20260928 14:38:24.611162 19539 raft_consensus.cc:2781] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:24.611442 19539 raft_consensus.cc:686] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [term 1 LEADER]: Becoming Leader. State: Replica: c88a069b086e446aaa93ecbb02de91c0, State: Running, Role: LEADER
I20260928 14:38:24.611788 19539 consensus_queue.cc:237] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } }
I20260928 14:38:24.615024 19033 catalog_manager.cc:5495] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c88a069b086e446aaa93ecbb02de91c0 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "c88a069b086e446aaa93ecbb02de91c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c88a069b086e446aaa93ecbb02de91c0" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 39527 } health_report { overall_health: HEALTHY } } }
I20260928 14:38:24.632203 19531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.628375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.6.23.193:39527)
0928 14:38:24.628507 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:24.628512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:24.628528 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:38:24.628620 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:24.628623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:24.628678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:24.628794 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:38:24.629059 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.629582 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.630463 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:24.630701 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:24.630812 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:24.631129 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:38:24.631767 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:38:24.631930 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:24.631964 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 14:38:24.632020 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:38:24.637753 19233 tablet.cc:2369] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:38:24.638952 19312 raft_consensus.cc:1261] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 1 FOLLOWER]: Refusing update from remote peer c88a069b086e446aaa93ecbb02de91c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:24.638959 19454 raft_consensus.cc:1261] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 1 FOLLOWER]: Refusing update from remote peer c88a069b086e446aaa93ecbb02de91c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:24.639789 19539 consensus_queue.cc:1035] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8da0c0df0fa845b6bc7233e019fc5d03" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:24.639981 19544 consensus_queue.cc:1035] T 72e5e896ea334dbdbe0b88f69b994ec5 P c88a069b086e446aaa93ecbb02de91c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "102d72dd2c864c37a93d1201d820d7f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:24.648118 19547 mvcc.cc:205] Tried to move back new op lower bound from 7334323423792599040 to 7334323423692984320. Current Snapshot: MvccSnapshot[applied={T|T < 7334323423792599040}]
I20260928 14:38:24.655851 19548 mvcc.cc:205] Tried to move back new op lower bound from 7334323423792599040 to 7334323423692984320. Current Snapshot: MvccSnapshot[applied={T|T < 7334323423792599040}]
I20260928 14:38:24.658535 19549 mvcc.cc:205] Tried to move back new op lower bound from 7334323423792599040 to 7334323423692984320. Current Snapshot: MvccSnapshot[applied={T|T < 7334323423792599040}]
I20260928 14:38:24.665503 19033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:38:24.665719 19033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:38:24.668706 19033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f62808a931044d2bb68b799a290f7a2e: Sending DeleteTablet for 3 replicas of tablet 72e5e896ea334dbdbe0b88f69b994ec5
I20260928 14:38:24.670185 19281 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e5e896ea334dbdbe0b88f69b994ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:24 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:52618
I20260928 14:38:24.670732 19434 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e5e896ea334dbdbe0b88f69b994ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:24 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:43956
I20260928 14:38:24.671658 19166 tablet_service.cc:1514] Processing DeleteTablet for tablet 72e5e896ea334dbdbe0b88f69b994ec5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:24 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:55962
I20260928 14:38:24.672726 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19072
I20260928 14:38:24.674266 19557 tablet_replica.cc:330] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9: stopping tablet replica
I20260928 14:38:24.674592 19557 raft_consensus.cc:2229] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:38:24.674860 19557 raft_consensus.cc:2258] T 72e5e896ea334dbdbe0b88f69b994ec5 P 102d72dd2c864c37a93d1201d820d7f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:24.685770 19558 tablet_replica.cc:330] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: stopping tablet replica
I20260928 14:38:24.686184 19558 raft_consensus.cc:2229] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:38:24.686551 19558 raft_consensus.cc:2258] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:24.688789 19558 ts_tablet_manager.cc:1905] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:24.691557 19558 ts_tablet_manager.cc:1918] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:24.691649 19558 log.cc:1196] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/72e5e896ea334dbdbe0b88f69b994ec5
I20260928 14:38:24.691936 19558 ts_tablet_manager.cc:1939] T 72e5e896ea334dbdbe0b88f69b994ec5 P 8da0c0df0fa845b6bc7233e019fc5d03: Deleting consensus metadata
I20260928 14:38:24.693580 19021 catalog_manager.cc:4841] TS 8da0c0df0fa845b6bc7233e019fc5d03 (127.6.23.195:43917): tablet 72e5e896ea334dbdbe0b88f69b994ec5 (table test-table [id=30b65517e6814794a3da48255d7b0252]) successfully deleted
I20260928 14:38:24.697862 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19236
W20260928 14:38:24.698535 19020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:39527 (error 108)
I20260928 14:38:24.700925 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.700564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:39527 (local address 127.0.0.1:55984)
0928 14:38:24.700764 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:24.700833 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:39527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260928 14:38:24.701220 19020 catalog_manager.cc:4584] TS c88a069b086e446aaa93ecbb02de91c0 (127.6.23.193:39527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72e5e896ea334dbdbe0b88f69b994ec5: Network error: Client connection negotiation failed: client connection to 127.6.23.193:39527: connect: Connection refused (error 111)
I20260928 14:38:24.707546 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19370
I20260928 14:38:24.708626 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:24.707702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:33109 (local address 127.0.0.1:52628)
0928 14:38:24.708492 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:24.708547 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.194:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":686}
I20260928 14:38:24.714269 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19001
I20260928 14:38:24.721568 6239 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:38:25.056597 6239 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:38:25.056718 6239 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:38:25.402062 6239 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:38:25.402 UTC [18068] LOG: received smart shutdown request
2026-09-28 14:38:25.406 UTC [18068] LOG: background worker "logical replication launcher" (PID 18077) exited with exit code 1
2026-09-28 14:38:25.406 UTC [18072] LOG: shutting down
2026-09-28 14:38:25.584 UTC [18068] LOG: database system is shut down
2026-09-28T14:38:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48859 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:38:25 dist-test-slave-06lh krb5kdc[19569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:38:25 dist-test-slave-06lh krb5kdc[19569](info): set up 2 sockets
Sep 28 14:38:25 dist-test-slave-06lh krb5kdc[19569](info): commencing operation
krb5kdc: starting...
W20260928 14:38:26.751618 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:38:26 dist-test-slave-06lh krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:38:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:38:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:38:26.878526 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:41047
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:41047,127.6.23.253:34861,127.6.23.252:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:26.992905 19582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:26.993239 19582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:26.993284 19582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:26.997437 19582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:26.997512 19582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:26.997536 19582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:26.997553 19582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:26.997568 19582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:27.003132 19582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:41047,127.6.23.253:34861,127.6.23.252:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:41047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:27.004288 19582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:27.005309 19582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:27.014211 19588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:27.014267 19582 server_base.cc:1034] running on GCE node
W20260928 14:38:27.014302 19590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:27.014213 19587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:27.015036 19582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:27.016332 19582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:27.017531 19582 hybrid_clock.cc:648] HybridClock initialized: now 1790606307017518 us; error 28 us; skew 500 ppm
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:27.024224 19582 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:38:27.025318 19582 webserver.cc:460] Webserver started at http://127.6.23.254:43477/ using document root <none> and password file <none>
I20260928 14:38:27.025906 19582 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:27.025971 19582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:27.026144 19582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:27.028415 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "827f1a484af74479ad2af61a2f7e8163"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "7813b858a6cbf50cbc482e44ee5f696a"
server_key_iv: "3cc1bea71894dabb80c6ec20d52940e6"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.028977 19582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "827f1a484af74479ad2af61a2f7e8163"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "7813b858a6cbf50cbc482e44ee5f696a"
server_key_iv: "3cc1bea71894dabb80c6ec20d52940e6"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.033078 19582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:38:27.035801 19597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:27.037218 19582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:38:27.037387 19582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "827f1a484af74479ad2af61a2f7e8163"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "7813b858a6cbf50cbc482e44ee5f696a"
server_key_iv: "3cc1bea71894dabb80c6ec20d52940e6"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.037519 19582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:27.054210 19582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:27.057446 19582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:27.057678 19582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:27.065142 19582 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:41047
I20260928 14:38:27.065178 19649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:41047 every 8 connection(s)
I20260928 14:38:27.066169 19582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:38:27.069144 19650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:27.073009 19650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:38:27.075515 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19582
I20260928 14:38:27.075655 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:38:27.076001 6239 external_mini_cluster.cc:1428] Setting key 523992728ce1df269662046ec4754340
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.253@KRBTEST.COM" created.
I20260928 14:38:27.089452 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.076791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.6.23.254:41047)
0928 14:38:27.077143 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.077154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.077203 (+ 49us) server_negotiation.cc:407] Connection header received
0928 14:38:27.077697 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.077702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.077899 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.078110 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.079017 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.079997 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.081045 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.081438 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.081611 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.084310 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.084319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.084336 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.084356 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.085958 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.086525 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.086538 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.086540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.086647 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.087085 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.087089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.087091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.087729 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.087879 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.088278 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.088582 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":237,"spinlock_wait_cycles":5504,"thread_start_us":114,"threads_started":1}
I20260928 14:38:27.089450 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.075584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41047 (local address 127.0.0.1:35824)
0928 14:38:27.076276 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.076325 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.077333 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.078110 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.078115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.078602 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.078861 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.078876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.080151 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.080154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.080921 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.080931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.081598 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.081602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.081715 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.082499 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.082540 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.084154 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.086077 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.086085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.086098 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.086407 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.086809 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.086813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.086818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.086970 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.087862 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.087868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.088008 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.088164 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.088464 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":206,"threads_started":1}
I20260928 14:38:27.092356 19650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:38:27.093191 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.092928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40766)
0928 14:38:27.093056 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.093127 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 14:38:27.093416 19600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
I20260928 14:38:27.094434 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.094180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40774)
0928 14:38:27.094304 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.094365 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 14:38:27.094584 19650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:34861: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab.
Entry for principal kudu/127.6.23.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab.
I20260928 14:38:27.106264 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.253:34861
--webserver_interface=127.6.23.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab
--principal=kudu/127.6.23.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:41047,127.6.23.253:34861,127.6.23.252:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:38:27.143836 19650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } attempt: 1
I20260928 14:38:27.144748 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.144427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40788)
0928 14:38:27.144580 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.144652 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:38:27.145845 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.145444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40798)
0928 14:38:27.145633 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.145742 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 14:38:27.146183 19650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:34861: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
I20260928 14:38:27.214411 19650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } attempt: 2
I20260928 14:38:27.215342 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.214992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40804)
0928 14:38:27.215138 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.215207 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 14:38:27.216158 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.215949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40814)
0928 14:38:27.216060 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.216107 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 14:38:27.216423 19650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:34861: Network error: Client connection negotiation failed: client connection to 127.6.23.253:34861: connect: Connection refused (error 111)
W20260928 14:38:27.224138 19657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:27.224424 19657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:27.224502 19657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:27.228786 19657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:27.228864 19657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:27.228884 19657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:27.228902 19657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:27.228941 19657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:27.234491 19657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:41047,127.6.23.253:34861,127.6.23.252:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab
--principal=kudu/127.6.23.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.253:34861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.23.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:27.235893 19657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:27.236999 19657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:27.246093 19662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:27.246275 19663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:27.246165 19657 server_base.cc:1034] running on GCE node
W20260928 14:38:27.246083 19665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:27.247061 19657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:27.248307 19657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:27.249475 19657 hybrid_clock.cc:648] HybridClock initialized: now 1790606307249458 us; error 30 us; skew 500 ppm
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:27.252532 19657 init.cc:377] Logged in from keytab as kudu/127.6.23.253@KRBTEST.COM (short username kudu)
I20260928 14:38:27.253516 19657 webserver.cc:460] Webserver started at http://127.6.23.253:44565/ using document root <none> and password file <none>
I20260928 14:38:27.254104 19657 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:27.254168 19657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:27.254419 19657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:27.256608 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/data/instance:
uuid: "8552bf319aae402ebbc9daa050170f9c"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "600c41d5024f11f9d5bbb91a1d169544"
server_key_iv: "5badfecf5b40d56763d35e665efc5c21"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.257195 19657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/wal/instance:
uuid: "8552bf319aae402ebbc9daa050170f9c"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "600c41d5024f11f9d5bbb91a1d169544"
server_key_iv: "5badfecf5b40d56763d35e665efc5c21"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.261624 19657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:38:27.264230 19672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:27.265537 19657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:38:27.265697 19657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/wal
uuid: "8552bf319aae402ebbc9daa050170f9c"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "600c41d5024f11f9d5bbb91a1d169544"
server_key_iv: "5badfecf5b40d56763d35e665efc5c21"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.265821 19657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:27.282984 19657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:27.286253 19657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:27.286484 19657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:27.294059 19657 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.253:34861
I20260928 14:38:27.294080 19724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.253:34861 every 8 connection(s)
I20260928 14:38:27.295115 19657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
I20260928 14:38:27.298250 19725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:27.302277 19725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:38:27.303421 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19657
I20260928 14:38:27.303553 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-1/wal/instance
I20260928 14:38:27.303843 6239 external_mini_cluster.cc:1428] Setting key 4a266bff28653bd3ff919330373cbf6e
I20260928 14:38:27.307628 19650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } attempt: 3
WARNING: no policy specified for kudu/127.6.23.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.252@KRBTEST.COM" created.
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:38:27.320888 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.308096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40820)
0928 14:38:27.308227 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.308244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.308322 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.309190 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.309193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.309418 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.309532 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.309539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.311110 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.311113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.311872 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.311884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.312729 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.312732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.312851 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.313247 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.313259 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.316607 (+ 3348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.319137 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.319143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.319145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.319413 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.319746 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.319752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.319757 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.319855 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.320385 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.320388 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.320480 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.320592 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.320721 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:38:27.321393 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.304463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.6.23.254:41047)
0928 14:38:27.304628 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.304633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.305568 (+ 935us) server_negotiation.cc:407] Connection header received
0928 14:38:27.306531 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.306537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.306605 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.306717 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.308586 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.309209 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.310518 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.310771 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.310890 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.316265 (+ 5375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.316270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.316276 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.316308 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.318127 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.318650 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.318655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.318656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.318736 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.319168 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.319174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.319179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.319456 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.319592 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.320999 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.321194 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:38:27.321722 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.304882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41047 (local address 127.0.0.1:35832)
0928 14:38:27.305386 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.305439 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.306304 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.306883 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.306891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.307285 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.307959 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.307986 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.309364 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.309367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.310392 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.310402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.310929 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.310933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.311140 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.311872 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.311893 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.316101 (+ 4208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.318242 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.318253 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.318265 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.318543 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.318828 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.318832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.318834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.319046 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.319606 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.319614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.319978 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.320210 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.320576 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":3,"thread_start_us":154,"threads_started":1}
I20260928 14:38:27.321913 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.308200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.6.23.253:34861)
0928 14:38:27.308610 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.308629 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.308655 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:38:27.308707 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.308715 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.308952 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.309202 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.309662 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.310996 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.312007 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.312585 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.312695 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.316808 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.316818 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.316840 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.316864 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.319005 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.319527 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.319533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.319535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.319646 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.319968 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.319971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.319973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.320258 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.321476 (+ 1218us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.321581 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.321721 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":284,"thread_start_us":178,"threads_started":1}
I20260928 14:38:27.323217 19725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:38:27.324084 19650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:38:27.324784 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.324509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32816)
0928 14:38:27.324639 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.324705 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:38:27.325688 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.325368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32830)
0928 14:38:27.325539 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.325594 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 14:38:27.325811 19650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:44523: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:38:27.331468 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.323846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.6.23.253:34861)
0928 14:38:27.324162 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.324166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.324179 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:38:27.324209 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.324211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.324250 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.324320 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.324754 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.325320 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.326220 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.326469 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.326575 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.328533 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.328536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.328539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.328560 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.329842 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.330280 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.330286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.330288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.330363 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.330620 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.330623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.330625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.330794 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.330897 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.331121 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.331293 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":247}
I20260928 14:38:27.331468 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.323834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40824)
0928 14:38:27.323998 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.324013 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.324085 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.324347 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.324349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.324513 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.324635 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.324643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.325429 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.325435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.326108 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.326117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.326583 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.326586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.326680 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.327085 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.327096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.328404 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.329988 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.329992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.329994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.330185 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.330453 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.330457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.330459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.330531 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.330885 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.330888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.330970 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.331069 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.331211 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 14:38:27.332291 19725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:38:27.333029 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.332769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32834)
0928 14:38:27.332882 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.332971 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:38:27.333603 19675 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab.
Entry for principal kudu/127.6.23.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab.
I20260928 14:38:27.334519 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.334278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32842)
0928 14:38:27.334396 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.334468 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 14:38:27.334692 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.252:44523
--webserver_interface=127.6.23.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab
--principal=kudu/127.6.23.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:34429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:41047,127.6.23.253:34861,127.6.23.252:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:27.334685 19725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:44523: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
I20260928 14:38:27.357009 19650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } attempt: 1
I20260928 14:38:27.357991 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.357636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32846)
0928 14:38:27.357816 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.357891 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 14:38:27.358748 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.358522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32854)
0928 14:38:27.358629 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.358680 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260928 14:38:27.358908 19650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:44523: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
I20260928 14:38:27.383929 19725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } attempt: 1
I20260928 14:38:27.384843 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.384493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32856)
0928 14:38:27.384671 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.384748 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 14:38:27.385619 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.385392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32868)
0928 14:38:27.385498 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.385551 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 14:38:27.385754 19725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:44523: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
I20260928 14:38:27.434098 19650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } attempt: 2
I20260928 14:38:27.435014 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.434665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32880)
0928 14:38:27.434840 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.434915 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 14:38:27.435814 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.435584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32884)
0928 14:38:27.435697 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.435752 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:38:27.435948 19650 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:44523: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
W20260928 14:38:27.452796 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:27.453104 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:27.453147 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:38:27.453902 19725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } attempt: 2
I20260928 14:38:27.454807 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.454474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32890)
0928 14:38:27.454642 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.454713 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 14:38:27.455667 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.455433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32892)
0928 14:38:27.455545 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.455599 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 14:38:27.455844 19725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:44523: Network error: Client connection negotiation failed: client connection to 127.6.23.252:44523: connect: Connection refused (error 111)
W20260928 14:38:27.458053 19732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:27.458119 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:27.458140 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:27.458159 19732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:27.458220 19732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:27.464110 19732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:41047,127.6.23.253:34861,127.6.23.252:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab
--principal=kudu/127.6.23.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.252:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.23.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:27.465288 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:27.466375 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:38:27.475510 19732 server_base.cc:1034] running on GCE node
W20260928 14:38:27.475525 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:27.475540 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:27.475526 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:27.476370 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:27.477504 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:27.478689 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790606307478660 us; error 46 us; skew 500 ppm
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:27.481668 19732 init.cc:377] Logged in from keytab as kudu/127.6.23.252@KRBTEST.COM (short username kudu)
I20260928 14:38:27.482620 19732 webserver.cc:460] Webserver started at http://127.6.23.252:43217/ using document root <none> and password file <none>
I20260928 14:38:27.483215 19732 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:27.483371 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:27.483624 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:27.485731 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/data/instance:
uuid: "a55a81f3f6934cb980f125d0ee341b44"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "2d50e2bf70eaa6234ecf16abcb52fde9"
server_key_iv: "aabd1deea16b351ce4a4c9bbbe4b3b1a"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.486311 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/wal/instance:
uuid: "a55a81f3f6934cb980f125d0ee341b44"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "2d50e2bf70eaa6234ecf16abcb52fde9"
server_key_iv: "aabd1deea16b351ce4a4c9bbbe4b3b1a"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.490085 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 14:38:27.492707 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:27.493992 19732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:38:27.494134 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/wal
uuid: "a55a81f3f6934cb980f125d0ee341b44"
format_stamp: "Formatted at 2026-09-28 14:38:27 on dist-test-slave-06lh"
server_key: "2d50e2bf70eaa6234ecf16abcb52fde9"
server_key_iv: "aabd1deea16b351ce4a4c9bbbe4b3b1a"
server_key_version: "encryptionkey@0"
I20260928 14:38:27.494251 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:27.513821 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:27.517513 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:27.517750 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:27.525691 19732 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.252:44523
I20260928 14:38:27.525709 19799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.252:44523 every 8 connection(s)
I20260928 14:38:27.526731 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
I20260928 14:38:27.529700 19800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:27.531571 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19732
I20260928 14:38:27.531675 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606142807864-6239-0/minicluster-data/master-2/wal/instance
I20260928 14:38:27.531977 6239 external_mini_cluster.cc:1428] Setting key 077ac8955ac08c0964e53c81e178d7c3
I20260928 14:38:27.534346 19800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:38:27.535125 19650 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } attempt: 3
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
I20260928 14:38:27.547086 19725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } attempt: 3
I20260928 14:38:27.548130 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.535628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32918)
0928 14:38:27.535795 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.535813 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.535908 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.536723 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.536726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.536891 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.537014 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.537021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.538388 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.538393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.539219 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.539256 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.540297 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.540301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.540396 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.541099 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.541120 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.542800 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.545469 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.545473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.545475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.545830 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.546222 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.546229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.546233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.546336 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.547505 (+ 1169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.547521 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.547636 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.547790 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.547941 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 14:38:27.548779 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.537115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.6.23.254:41047)
0928 14:38:27.537219 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.537224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.537593 (+ 369us) server_negotiation.cc:407] Connection header received
0928 14:38:27.537764 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.537767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.537811 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.537896 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.538725 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.539195 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.540211 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.540520 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.540620 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.543999 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.544005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.544009 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.544039 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.545936 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.547095 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.547099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.547101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.547190 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.547638 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.547642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.547644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.547892 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.547976 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.548492 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.548625 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:38:27.550129 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.537060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:41047 (local address 127.0.0.1:35842)
0928 14:38:27.537474 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.537525 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.537672 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.538050 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.538056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.538500 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.538608 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.538614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.539375 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.539378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.540086 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.540097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.540718 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.540723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.540869 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.541339 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.541355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.543811 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.546781 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.546788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.546793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.546984 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.547407 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.547413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.547415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.547504 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.548047 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.548052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.548222 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.548986 (+ 764us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.549189 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":26,"thread_start_us":79,"threads_started":1}
I20260928 14:38:27.550199 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.536069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.6.23.252:44523)
0928 14:38:27.536477 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.536486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.536499 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:38:27.536531 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.536534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.536570 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.537128 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.537236 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.538226 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.539405 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.540172 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.540667 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.542956 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.542961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.542971 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.542992 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.545330 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.545964 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.545973 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.545981 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.546127 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.546434 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.546436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.546438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.546973 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.548468 (+ 1495us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.548606 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.548945 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":228,"server-negotiator.queue_time_us":340,"thread_start_us":250,"threads_started":1}
I20260928 14:38:27.550732 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.533468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.6.23.252:44523)
0928 14:38:27.533924 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.533945 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.533990 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:38:27.534799 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.534834 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.535312 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.535684 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.536671 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.538200 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.539405 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.540157 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.540294 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.544271 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.544275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.544278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.544304 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.546701 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.547211 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.547217 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.547221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.547347 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.548727 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.548731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.548733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.549167 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.549942 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.550018 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.550129 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":293,"thread_start_us":109,"threads_started":1}
I20260928 14:38:27.552145 19800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
I20260928 14:38:27.555536 19650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163: Bootstrap starting.
I20260928 14:38:27.558353 19650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:27.559214 19727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.547629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32922)
0928 14:38:27.547836 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.547851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.547930 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.548633 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.548636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.548819 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.548950 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.548958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.550048 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.550052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.550851 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.550862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.551479 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.551483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.551566 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.552096 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.552110 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.553742 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.557241 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.557244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.557247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.557566 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.557924 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.557927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.557930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.558032 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.558659 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.558664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.558784 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.558874 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.559028 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260928 14:38:27.559386 19650 log.cc:824] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:27.560025 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.547699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.6.23.252:44523)
0928 14:38:27.548382 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.548393 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.548406 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:38:27.548439 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.548441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.548489 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.549073 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.549160 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.549906 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.550988 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.551352 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.552215 (+ 863us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.553874 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.553879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.553883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.553912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.556377 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.557708 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.557712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.557714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.557809 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.558158 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.558164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.558169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.558525 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.559602 (+ 1077us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.559680 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.559831 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":112,"threads_started":1}
I20260928 14:38:27.563360 19650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163: No bootstrap required, opened a new log
I20260928 14:38:27.563539 19725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: Bootstrap starting.
I20260928 14:38:27.564287 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.552885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.6.23.253:34861)
0928 14:38:27.552995 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.552999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.553537 (+ 538us) server_negotiation.cc:407] Connection header received
0928 14:38:27.553696 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.553700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.553741 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.553793 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.554730 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.555467 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.557246 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.557576 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.557676 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.560677 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.560683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.560689 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.560718 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.562475 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.562939 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.562942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.562944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.563035 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.563371 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.563374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.563376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.563609 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.563682 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.564037 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.564150 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:38:27.564571 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.553213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.6.23.254:41047)
0928 14:38:27.553326 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.553330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.553344 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:38:27.553389 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.553392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.553433 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.553507 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.554340 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.555002 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.556687 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.557006 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.557124 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.560585 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.560589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.560591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.560623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.562329 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.562835 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.562838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.562840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.562924 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.563260 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.563263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.563264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.563501 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.563677 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.564310 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.564426 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:38:27.565079 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.552642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:34861 (local address 127.0.0.1:40836)
0928 14:38:27.553474 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.553489 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.553595 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.553984 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.553988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.554467 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.554595 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.554603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.555582 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.555587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.557058 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.557067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.557788 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.557791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.557891 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.558324 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.558338 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.560525 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.562596 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.562599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.562601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.562830 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.563145 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.563148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.563149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.563260 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.563755 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.563758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.563851 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.564445 (+ 594us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.564593 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753}
I20260928 14:38:27.565732 19800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:38:27.567165 19725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:27.567802 19650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.568073 19650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:27.568140 19650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827f1a484af74479ad2af61a2f7e8163, State: Initialized, Role: FOLLOWER
I20260928 14:38:27.568467 19725 log.cc:824] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:27.568650 19650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.570127 19650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:27.570986 19725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: No bootstrap required, opened a new log
I20260928 14:38:27.570526 19809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.571835 19809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606307, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
Sep 28 14:38:27 dist-test-slave-06lh krb5kdc[19569](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:38:27.575701 19725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.576009 19725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:27.576118 19725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8552bf319aae402ebbc9daa050170f9c, State: Initialized, Role: FOLLOWER
I20260928 14:38:27.576663 19725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.576957 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.566167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:44523 (local address 127.0.0.1:32930)
0928 14:38:27.567424 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:27.567439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:27.567529 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:27.567802 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:27.567805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:27.567964 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:27.568058 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.568065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.568991 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.568995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.569801 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:27.569812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.570449 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.570455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:27.570566 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.571036 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:27.571049 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:27.573314 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:27.575377 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.575380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.575382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.575626 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.575909 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:27.575914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:27.575916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:27.575998 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:27.576423 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:27.576426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:27.576533 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 14:38:27.576635 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:38:27.576771 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"thread_start_us":122,"threads_started":1}
I20260928 14:38:27.577457 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.566500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.6.23.252:44523)
0928 14:38:27.566641 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.566645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.567598 (+ 953us) server_negotiation.cc:407] Connection header received
0928 14:38:27.567623 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.567626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.567658 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.567713 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.568179 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.568870 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.569927 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.570234 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.570344 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.573489 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.573494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.573497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.573520 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.575266 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.575722 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.575725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.575728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.575816 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.576087 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.576089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.576091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.576308 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.576392 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.577203 (+ 811us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.577323 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:38:27.577726 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.566005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.6.23.253:34861)
0928 14:38:27.566117 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.566121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.566418 (+ 297us) server_negotiation.cc:407] Connection header received
0928 14:38:27.566458 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.566461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.566505 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.566574 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.567514 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.568293 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.569425 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.569748 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.569843 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.573570 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.573574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.573577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.573602 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.575404 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.575902 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.575908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.575913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.575998 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.576328 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.576331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.576332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.576536 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.576696 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.577446 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.577567 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:38:27.579176 19725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:38:27.580518 19821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:38:27.580596 19821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:38:27.580576 19800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: Bootstrap starting.
I20260928 14:38:27.581748 19822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.581909 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:38:27.581944 19822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:38:27.583354 19800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:27.584295 19800 log.cc:824] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:27.577241 19582 master_runner.cc:186] Error getting master registration for 127.6.23.253:34861: OK, instance_id { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" instance_seqno: 1790606307291721 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:38:27.585855 19582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:38:27.586544 19657 master_runner.cc:186] Error getting master registration for 127.6.23.252:44523: OK, instance_id { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" instance_seqno: 1790606307523427 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:38:27.590380 19657 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:38:27.593071 19800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: No bootstrap required, opened a new log
W20260928 14:38:27.590391 19834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:38:27.595366 19834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:38:27.596150 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.586288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.6.23.253:34861)
0928 14:38:27.586507 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.586523 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.586650 (+ 127us) server_negotiation.cc:407] Connection header received
0928 14:38:27.586843 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.586847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.586896 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.586957 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.588080 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.588856 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.590010 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.590346 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.590414 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.592467 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.592471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.592474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.592499 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.594154 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.594661 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.594665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.594667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.594753 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.595058 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.595061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.595063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.595314 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.595420 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.595878 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.596001 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260928 14:38:27.597447 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.586698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.6.23.252:44523)
0928 14:38:27.586813 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.586819 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.587363 (+ 544us) server_negotiation.cc:407] Connection header received
0928 14:38:27.587548 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.587553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.587610 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.587667 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.588552 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.589309 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.590328 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.590650 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.590717 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.593908 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.593912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.593914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.593939 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.595599 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.596091 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.596094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.596097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.596180 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.596483 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.596488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.596489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.596689 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.596811 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.597175 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.597297 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:38:27.599881 19800 raft_consensus.cc:348] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.600159 19800 raft_consensus.cc:374] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:27.600234 19800 raft_consensus.cc:729] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55a81f3f6934cb980f125d0ee341b44, State: Initialized, Role: FOLLOWER
I20260928 14:38:27.600847 19800 consensus_queue.cc:260] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.601703 19844 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.601898 19844 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:38:27.602527 19800 sys_catalog.cc:564] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:27.606781 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:27.586004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.6.23.254:41047)
0928 14:38:27.586120 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:27.586125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:27.596941 (+ 10816us) server_negotiation.cc:407] Connection header received
0928 14:38:27.597127 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:27.597131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:27.597187 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:27.597240 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:27.598395 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.599204 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.600304 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:27.600652 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:27.600730 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:27.602516 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:27.602520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:27.602524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:27.602552 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:27.604483 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.605013 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.605016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.605018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.605109 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:27.605413 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:27.605420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:27.605425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:27.605661 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:27.605799 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:27.606469 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 14:38:27.606615 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 14:38:27.609258 19855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:38:27.609359 19855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:38:27.610777 19732 master_runner.cc:263] No leader master found from master a55a81f3f6934cb980f125d0ee341b44
I20260928 14:38:27.610857 19732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 14:38:27.769248 19753 tablet.cc:2369] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:38:27.788169 19678 tablet.cc:2369] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:38:27.809593 19603 tablet.cc:2369] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:38:27.812402 19809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:27.812642 19809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.813633 19809 leader_election.cc:288] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8552bf319aae402ebbc9daa050170f9c (127.6.23.253:34861), a55a81f3f6934cb980f125d0ee341b44 (127.6.23.252:44523)
I20260928 14:38:27.814303 19774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827f1a484af74479ad2af61a2f7e8163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55a81f3f6934cb980f125d0ee341b44" is_pre_election: true
I20260928 14:38:27.814306 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827f1a484af74479ad2af61a2f7e8163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8552bf319aae402ebbc9daa050170f9c" is_pre_election: true
I20260928 14:38:27.814643 19699 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827f1a484af74479ad2af61a2f7e8163 in term 0.
I20260928 14:38:27.814651 19774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827f1a484af74479ad2af61a2f7e8163 in term 0.
I20260928 14:38:27.815081 19600 leader_election.cc:302] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827f1a484af74479ad2af61a2f7e8163, 8552bf319aae402ebbc9daa050170f9c; no voters:
I20260928 14:38:27.815449 19809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:38:27.815574 19809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:38:27.815639 19809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:27.817116 19809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.817526 19809 leader_election.cc:288] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [CANDIDATE]: Term 1 election: Requested vote from peers 8552bf319aae402ebbc9daa050170f9c (127.6.23.253:34861), a55a81f3f6934cb980f125d0ee341b44 (127.6.23.252:44523)
I20260928 14:38:27.817971 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827f1a484af74479ad2af61a2f7e8163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8552bf319aae402ebbc9daa050170f9c"
I20260928 14:38:27.818033 19774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "827f1a484af74479ad2af61a2f7e8163" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a55a81f3f6934cb980f125d0ee341b44"
I20260928 14:38:27.818156 19774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:27.818140 19699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:27.819614 19699 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827f1a484af74479ad2af61a2f7e8163 in term 1.
I20260928 14:38:27.819617 19774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827f1a484af74479ad2af61a2f7e8163 in term 1.
I20260928 14:38:27.819978 19600 leader_election.cc:302] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827f1a484af74479ad2af61a2f7e8163, 8552bf319aae402ebbc9daa050170f9c; no voters:
I20260928 14:38:27.820216 19809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:27.820523 19809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [term 1 LEADER]: Becoming Leader. State: Replica: 827f1a484af74479ad2af61a2f7e8163, State: Running, Role: LEADER
I20260928 14:38:27.820907 19809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } }
I20260928 14:38:27.823212 19857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827f1a484af74479ad2af61a2f7e8163. Latest consensus state: current_term: 1 leader_uuid: "827f1a484af74479ad2af61a2f7e8163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.823415 19857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:27.823909 19859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:38:27.824783 19859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:38:27.829478 19774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [term 1 FOLLOWER]: Refusing update from remote peer 827f1a484af74479ad2af61a2f7e8163: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:27.829491 19699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [term 1 FOLLOWER]: Refusing update from remote peer 827f1a484af74479ad2af61a2f7e8163: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:27.830196 19857 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:27.830380 19809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:27.834785 19822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827f1a484af74479ad2af61a2f7e8163. Latest consensus state: current_term: 1 leader_uuid: "827f1a484af74479ad2af61a2f7e8163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.834833 19844 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827f1a484af74479ad2af61a2f7e8163. Latest consensus state: current_term: 1 leader_uuid: "827f1a484af74479ad2af61a2f7e8163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.834939 19822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:38:27.834957 19844 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:38:27.837455 19862 mvcc.cc:205] Tried to move back new op lower bound from 7334323436862918656 to 7334323436837220352. Current Snapshot: MvccSnapshot[applied={T|T < 7334323436862918656}]
I20260928 14:38:27.837471 19857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "827f1a484af74479ad2af61a2f7e8163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.837611 19857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:27.837718 19857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "827f1a484af74479ad2af61a2f7e8163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.837797 19857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163 [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:27.838346 19822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "827f1a484af74479ad2af61a2f7e8163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.838476 19822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:38:27.838840 19844 sys_catalog.cc:455] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "827f1a484af74479ad2af61a2f7e8163" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "827f1a484af74479ad2af61a2f7e8163" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41047 } } peers { permanent_uuid: "8552bf319aae402ebbc9daa050170f9c" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 34861 } } peers { permanent_uuid: "a55a81f3f6934cb980f125d0ee341b44" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 44523 } } }
I20260928 14:38:27.838963 19844 sys_catalog.cc:458] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:38:27.839067 19859 catalog_manager.cc:1347] Generated new cluster ID: e88c0d67d9f043f897bd85ae4f749a9f
I20260928 14:38:27.839131 19859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:38:27.848398 19859 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:38:27.849597 19859 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:38:27.857755 19859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 827f1a484af74479ad2af61a2f7e8163: Generated new TSK 0
I20260928 14:38:27.858711 19859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:38:28.596994 19834 catalog_manager.cc:1259] Loaded cluster ID: e88c0d67d9f043f897bd85ae4f749a9f
I20260928 14:38:28.597100 19834 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: loading cluster ID for follower catalog manager: success
I20260928 14:38:28.599078 19834 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: acquiring CA information for follower catalog manager: success
I20260928 14:38:28.600354 19834 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8552bf319aae402ebbc9daa050170f9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:38:28.610646 19855 catalog_manager.cc:1259] Loaded cluster ID: e88c0d67d9f043f897bd85ae4f749a9f
I20260928 14:38:28.610757 19855 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: loading cluster ID for follower catalog manager: success
I20260928 14:38:28.612767 19855 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: acquiring CA information for follower catalog manager: success
I20260928 14:38:28.613962 19855 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a55a81f3f6934cb980f125d0ee341b44: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:38:28.920033 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19582
I20260928 14:38:28.927065 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19657
I20260928 14:38:28.933264 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19732
2026-09-28T14:38:28Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3225 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:38:28 dist-test-slave-06lh krb5kdc[19875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:38:28 dist-test-slave-06lh krb5kdc[19875](info): set up 2 sockets
Sep 28 14:38:28 dist-test-slave-06lh krb5kdc[19875](info): commencing operation
krb5kdc: starting...
W20260928 14:38:29.979820 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:38:30 dist-test-slave-06lh krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:38:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:38:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:38:30.101897 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:41795
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:41795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:30.217326 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:30.217653 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:30.217696 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:30.221858 19888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:30.221937 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:30.221958 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:30.221982 19888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:30.221998 19888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:30.227634 19888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:41795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:41795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:30.228740 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:30.229744 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:30.238448 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:30.238452 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:30.238448 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:30.238701 19888 server_base.cc:1034] running on GCE node
I20260928 14:38:30.239481 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:30.240672 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:30.241842 19888 hybrid_clock.cc:648] HybridClock initialized: now 1790606310241830 us; error 27 us; skew 500 ppm
Sep 28 14:38:30 dist-test-slave-06lh krb5kdc[19875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606310, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:30.244760 19888 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:38:30.245740 19888 webserver.cc:460] Webserver started at http://127.6.23.254:42173/ using document root <none> and password file <none>
I20260928 14:38:30.246287 19888 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:30.246349 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:30.246527 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:30.249253 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "d2203b5770c34bcf94f3a314cef115ef"
format_stamp: "Formatted at 2026-09-28 14:38:30 on dist-test-slave-06lh"
server_key: "a2400185a543358cffafee5a690eec29"
server_key_iv: "344237ef2eb2c99c0edbe14919178f73"
server_key_version: "encryptionkey@0"
I20260928 14:38:30.249963 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "d2203b5770c34bcf94f3a314cef115ef"
format_stamp: "Formatted at 2026-09-28 14:38:30 on dist-test-slave-06lh"
server_key: "a2400185a543358cffafee5a690eec29"
server_key_iv: "344237ef2eb2c99c0edbe14919178f73"
server_key_version: "encryptionkey@0"
I20260928 14:38:30.253970 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:38:30.256650 19903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:30.257956 19888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:38:30.258116 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "d2203b5770c34bcf94f3a314cef115ef"
format_stamp: "Formatted at 2026-09-28 14:38:30 on dist-test-slave-06lh"
server_key: "a2400185a543358cffafee5a690eec29"
server_key_iv: "344237ef2eb2c99c0edbe14919178f73"
server_key_version: "encryptionkey@0"
I20260928 14:38:30.258242 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:30.270519 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:30.273764 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:30.273993 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:30.281757 19888 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:41795
I20260928 14:38:30.281766 19955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:41795 every 8 connection(s)
I20260928 14:38:30.282784 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:38:30.285773 19956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:30.288969 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19888
I20260928 14:38:30.289116 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:38:30.289436 6239 external_mini_cluster.cc:1428] Setting key 886a2baf8f691fa6d585c4704324c603
I20260928 14:38:30.292279 19956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef: Bootstrap starting.
I20260928 14:38:30.294520 19956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:30.295362 19956 log.cc:824] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:30.297371 19956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef: No bootstrap required, opened a new log
Sep 28 14:38:30 dist-test-slave-06lh krb5kdc[19875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:30.300771 19956 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2203b5770c34bcf94f3a314cef115ef" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41795 } }
I20260928 14:38:30.301054 19956 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:30.301149 19956 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2203b5770c34bcf94f3a314cef115ef, State: Initialized, Role: FOLLOWER
I20260928 14:38:30.301672 19956 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2203b5770c34bcf94f3a314cef115ef" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41795 } }
I20260928 14:38:30.301862 19956 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:38:30.301970 19956 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:38:30.302094 19956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:30.303143 19956 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2203b5770c34bcf94f3a314cef115ef" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41795 } }
I20260928 14:38:30.303544 19956 leader_election.cc:302] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2203b5770c34bcf94f3a314cef115ef; no voters:
I20260928 14:38:30.303890 19956 leader_election.cc:288] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:38:30.303994 19961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:30.304234 19961 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [term 1 LEADER]: Becoming Leader. State: Replica: d2203b5770c34bcf94f3a314cef115ef, State: Running, Role: LEADER
I20260928 14:38:30.304623 19961 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2203b5770c34bcf94f3a314cef115ef" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41795 } }
I20260928 14:38:30.304852 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:30.290913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.6.23.254:41795)
0928 14:38:30.291429 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:30.291444 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:30.291483 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:38:30.292125 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:30.292141 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:30.292539 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:30.292882 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:30.293692 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:30.294763 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:30.295728 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:30.296210 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:30.296384 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:30.298920 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:30.298925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:30.298939 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:30.298965 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:30.301364 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:30.301900 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:30.301908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:30.301913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:30.302085 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:30.302407 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:30.302409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:30.302411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:30.302809 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:30.302966 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:30.303387 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 14:38:30.303749 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":161,"threads_started":1}
I20260928 14:38:30.305266 19956 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:30.305934 19961 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2203b5770c34bcf94f3a314cef115ef. Latest consensus state: current_term: 1 leader_uuid: "d2203b5770c34bcf94f3a314cef115ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2203b5770c34bcf94f3a314cef115ef" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41795 } } }
I20260928 14:38:30.306128 19961 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:30.306559 19970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:38:30.307327 19962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2203b5770c34bcf94f3a314cef115ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2203b5770c34bcf94f3a314cef115ef" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 41795 } } }
I20260928 14:38:30.307433 19962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:30.308203 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:38:30.309854 19970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:38:30.316257 19970 catalog_manager.cc:1347] Generated new cluster ID: edeae4999e204f6ea848101c22f9befa
I20260928 14:38:30.316357 19970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:38:30.332149 19970 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:38:30.333209 19970 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:38:30.341315 19970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2203b5770c34bcf94f3a314cef115ef: Generated new TSK 0
I20260928 14:38:30.342149 19970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:38:30.364625 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19888
2026-09-28T14:38:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1428 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:38:30 dist-test-slave-06lh krb5kdc[19983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:38:30 dist-test-slave-06lh krb5kdc[19983](info): set up 2 sockets
Sep 28 14:38:30 dist-test-slave-06lh krb5kdc[19983](info): commencing operation
krb5kdc: starting...
W20260928 14:38:31.403847 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:38:31 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:38:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:38:31.538821 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:38565
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:38565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:31.653870 19996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:31.654191 19996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:31.654232 19996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:31.658459 19996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:31.658542 19996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:31.658562 19996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:31.658581 19996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:31.658595 19996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:31.664254 19996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:38565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:38565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:31.665422 19996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:31.666576 19996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:31.675905 20001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:31.675948 20002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:31.675905 20004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:31.676194 19996 server_base.cc:1034] running on GCE node
I20260928 14:38:31.676918 19996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:31.678066 19996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:31.679381 19996 hybrid_clock.cc:648] HybridClock initialized: now 1790606311679355 us; error 34 us; skew 500 ppm
Sep 28 14:38:31 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606311, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:31.682125 19996 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:38:31.683070 19996 webserver.cc:460] Webserver started at http://127.6.23.254:37865/ using document root <none> and password file <none>
I20260928 14:38:31.683679 19996 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:31.683741 19996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:31.683909 19996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:31.686024 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "3a10326e298041d397dc104bff976c7a"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "d3678809c8b0d8fb0a57434a1d77ee22"
server_key_iv: "924c87b2efa8adabc93d3f1bd6514d64"
server_key_version: "encryptionkey@0"
I20260928 14:38:31.686600 19996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "3a10326e298041d397dc104bff976c7a"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "d3678809c8b0d8fb0a57434a1d77ee22"
server_key_iv: "924c87b2efa8adabc93d3f1bd6514d64"
server_key_version: "encryptionkey@0"
I20260928 14:38:31.690608 19996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:38:31.693311 20011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:31.694669 19996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:38:31.694830 19996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "3a10326e298041d397dc104bff976c7a"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "d3678809c8b0d8fb0a57434a1d77ee22"
server_key_iv: "924c87b2efa8adabc93d3f1bd6514d64"
server_key_version: "encryptionkey@0"
I20260928 14:38:31.694959 19996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:31.736171 19996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:31.739171 19996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:31.739405 19996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:31.746713 19996 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:38565
I20260928 14:38:31.747040 20063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:38565 every 8 connection(s)
I20260928 14:38:31.747874 19996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:38:31.750834 20064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:31.756642 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 19996
I20260928 14:38:31.756788 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:38:31.757081 6239 external_mini_cluster.cc:1428] Setting key f94da223e29af2d1207d6960375dc408
I20260928 14:38:31.757257 20064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Bootstrap starting.
I20260928 14:38:31.760067 20064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:31.760874 20064 log.cc:824] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:31.762872 20064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: No bootstrap required, opened a new log
Sep 28 14:38:31 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:31.767311 20064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:31.767609 20064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:31.767678 20064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a10326e298041d397dc104bff976c7a, State: Initialized, Role: FOLLOWER
I20260928 14:38:31.768191 20064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:31.768361 20064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:38:31.768455 20064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:38:31.768579 20064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:31.769551 20064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:31.769961 20064 leader_election.cc:302] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a10326e298041d397dc104bff976c7a; no voters:
I20260928 14:38:31.770365 20064 leader_election.cc:288] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:38:31.770484 20069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:31.770712 20069 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 1 LEADER]: Becoming Leader. State: Replica: 3a10326e298041d397dc104bff976c7a, State: Running, Role: LEADER
I20260928 14:38:31.771083 20069 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:31.771873 20064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:31.773038 20071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a10326e298041d397dc104bff976c7a. Latest consensus state: current_term: 1 leader_uuid: "3a10326e298041d397dc104bff976c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } } }
I20260928 14:38:31.773044 20070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a10326e298041d397dc104bff976c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } } }
I20260928 14:38:31.773188 20070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:31.773188 20071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:31.773391 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:31.758668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.6.23.254:38565)
0928 14:38:31.759147 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:31.759164 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:31.759200 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:38:31.759994 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:31.760013 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:31.760421 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:31.760768 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:31.761593 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:31.762581 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:31.763576 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:31.764283 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:31.764462 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:31.767169 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:31.767175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:31.767190 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:31.767221 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:31.769300 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:31.769770 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:31.769776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:31.769782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:31.769934 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:31.770278 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:31.770285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:31.770290 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:31.770892 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:31.771774 (+ 882us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:31.771944 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 14:38:31.772271 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":154,"threads_started":1}
I20260928 14:38:31.773958 20078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:38:31.777364 20078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:38:31.783411 20078 catalog_manager.cc:1347] Generated new cluster ID: ba768ef0a3694d9ab92ace02dbb61fd3
I20260928 14:38:31.783515 20078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:38:31.797227 20078 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:38:31.798365 20078 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:38:31.804527 20078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Generated new TSK 0
I20260928 14:38:31.805312 20078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:38:31.836278 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38565
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:31.953922 20090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:31.954216 20090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:31.954262 20090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:31.958464 20090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:31.958536 20090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:31.958626 20090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:38:31.964355 20090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:31.965560 20090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:31.966701 20090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:31.977375 20096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:31.977370 20095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:31.977370 20098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:31.977417 20090 server_base.cc:1034] running on GCE node
I20260928 14:38:31.978088 20090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:31.978753 20090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:31.979954 20090 hybrid_clock.cc:648] HybridClock initialized: now 1790606311979927 us; error 44 us; skew 500 ppm
Sep 28 14:38:31 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606311, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:31.982892 20090 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:38:31.983952 20090 webserver.cc:460] Webserver started at http://127.6.23.193:38933/ using document root <none> and password file <none>
I20260928 14:38:31.984576 20090 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:31.984642 20090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:31.984898 20090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:31.987354 20090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "18a24d3950cac92c09860790b7a88704"
server_key_iv: "d5069ee94f209acb5c004b93b19ccfbd"
server_key_version: "encryptionkey@0"
I20260928 14:38:31.988004 20090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "18a24d3950cac92c09860790b7a88704"
server_key_iv: "d5069ee94f209acb5c004b93b19ccfbd"
server_key_version: "encryptionkey@0"
I20260928 14:38:31.992169 20090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:38:31.994715 20105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:31.996029 20090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:38:31.996169 20090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "18a24d3950cac92c09860790b7a88704"
server_key_iv: "d5069ee94f209acb5c004b93b19ccfbd"
server_key_version: "encryptionkey@0"
I20260928 14:38:31.996284 20090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:32.023957 20090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:32.031720 20090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:32.031958 20090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:32.032620 20090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:32.033563 20090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:32.033632 20090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:32.033704 20090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:32.033743 20090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:32.043555 20090 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:33933
I20260928 14:38:32.043603 20218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:33933 every 8 connection(s)
I20260928 14:38:32.044545 20090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:38:32.053982 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 20090
I20260928 14:38:32.054129 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:38:32.054430 6239 external_mini_cluster.cc:1428] Setting key 328867137ae0e30623ac2dba9d82ad2e
Sep 28 14:38:32 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606311, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:32.059942 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.046483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:39195 (local address 127.6.23.254:38565)
0928 14:38:32.046625 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.046630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.047779 (+ 1149us) server_negotiation.cc:407] Connection header received
0928 14:38:32.048740 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.048744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.048803 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.048890 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:32.050162 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.050720 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.052019 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.052265 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.052383 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.055258 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:32.055262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:32.055265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:32.055296 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:32.057191 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:32.057762 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:32.057767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:32.057770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:32.057882 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:32.058403 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:32.058406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:32.058407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:32.058624 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:32.058705 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.059521 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.059684 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:38:32.060731 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.046855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:39195)
0928 14:38:32.047612 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.047666 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:32.048520 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:32.049092 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:32.049099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:32.049488 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:32.050001 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.050016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.050867 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.050871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.051874 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.051884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.052432 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.052435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.052613 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.053199 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:32.053215 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:32.055049 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:32.057325 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:32.057331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:32.057346 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:32.057652 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:32.058016 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:32.058023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:32.058026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:32.058256 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:32.058736 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:32.058746 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:32.059143 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 14:38:32.059429 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 14:38:32.059731 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":120,"threads_started":1}
I20260928 14:38:32.062126 20219 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38565
I20260928 14:38:32.062433 20219 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:32.063117 20219 heartbeater.cc:511] Master 127.6.23.254:38565 requested a full tablet report, sending...
I20260928 14:38:32.064882 20028 ts_manager.cc:194] Registered new tserver with Master: 585b9c9bfbb1447ea05b2db1f1bfbd2c (127.6.23.193:33933)
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:38:32.066169 20028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:39195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:38:32.082700 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38565
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:32.201290 20224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:32.201591 20224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:32.201660 20224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:32.205880 20224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:32.205964 20224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:32.206061 20224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:38:32.211730 20224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:32.212946 20224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:32.214059 20224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:32.224565 20230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:32.224731 20224 server_base.cc:1034] running on GCE node
W20260928 14:38:32.224599 20232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:32.224565 20229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:32.225471 20224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:32.226263 20224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:32.227453 20224 hybrid_clock.cc:648] HybridClock initialized: now 1790606312227410 us; error 59 us; skew 500 ppm
Sep 28 14:38:32 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606312, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:32.230391 20224 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:38:32.231433 20224 webserver.cc:460] Webserver started at http://127.6.23.194:40129/ using document root <none> and password file <none>
I20260928 14:38:32.232003 20224 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:32.232064 20224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:32.232232 20224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:32.234294 20224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "dabcee39ccd945ef92f4c9edd826d45c"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "87b6e196c9486f8d682742219f7c45a8"
server_key_iv: "9030694fdc42236cc4307bf8f04cd202"
server_key_version: "encryptionkey@0"
I20260928 14:38:32.234830 20224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "dabcee39ccd945ef92f4c9edd826d45c"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "87b6e196c9486f8d682742219f7c45a8"
server_key_iv: "9030694fdc42236cc4307bf8f04cd202"
server_key_version: "encryptionkey@0"
I20260928 14:38:32.238713 20224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:38:32.241307 20239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:32.242720 20224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:38:32.242820 20224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "dabcee39ccd945ef92f4c9edd826d45c"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "87b6e196c9486f8d682742219f7c45a8"
server_key_iv: "9030694fdc42236cc4307bf8f04cd202"
server_key_version: "encryptionkey@0"
I20260928 14:38:32.242910 20224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:32.258569 20224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:32.262063 20224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:32.262223 20224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:32.262780 20224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:32.263677 20224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:32.263726 20224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:32.263769 20224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:32.263787 20224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:32.273092 20224 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:37691
I20260928 14:38:32.273222 20352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:37691 every 8 connection(s)
I20260928 14:38:32.274251 20224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:38:32.279752 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 20224
I20260928 14:38:32.279887 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:38:32.280179 6239 external_mini_cluster.cc:1428] Setting key ad9ccbbce36245a7420d680bb5566f82
Sep 28 14:38:32 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606312, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:32.289075 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.276209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:38565 (local address 127.6.23.254:38565)
0928 14:38:32.276345 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.276349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.277456 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 14:38:32.278467 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.278470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.278517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.278593 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:32.279910 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.280575 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.282127 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.282349 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.282431 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.285414 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:32.285418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:32.285420 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:32.285443 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:32.286903 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:32.287522 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:32.287526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:32.287528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:32.287608 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:32.287961 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:32.287965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:32.287967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:32.288157 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:32.288263 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.288778 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.288910 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:38:32.290001 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.276636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:38565)
0928 14:38:32.277312 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.277356 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:32.278276 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:32.278767 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:32.278774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:32.279144 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:32.279753 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.279767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.280726 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.280730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.281948 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.281961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.282511 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.282517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.282768 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.283486 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:32.283512 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:32.285233 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:32.287030 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:32.287036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:32.287053 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:32.287415 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:32.287700 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:32.287704 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:32.287706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:32.287858 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:32.288260 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:32.288267 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:32.288537 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 14:38:32.288728 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 14:38:32.289086 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":113,"threads_started":1}
I20260928 14:38:32.291316 20353 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38565
I20260928 14:38:32.291576 20353 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:32.292122 20353 heartbeater.cc:511] Master 127.6.23.254:38565 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:38:32.293289 20028 ts_manager.cc:194] Registered new tserver with Master: dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691)
I20260928 14:38:32.293897 20028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:38565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:38:32.310743 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38565
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:32.426858 20358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:32.427168 20358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:32.427217 20358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:32.431667 20358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:32.431734 20358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:32.431798 20358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:38:32.437381 20358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:32.438488 20358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:32.439581 20358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:32.450145 20364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:32.450150 20366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:32.450177 20363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:32.450659 20358 server_base.cc:1034] running on GCE node
I20260928 14:38:32.451068 20358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:32.451805 20358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:32.452988 20358 hybrid_clock.cc:648] HybridClock initialized: now 1790606312452970 us; error 34 us; skew 500 ppm
Sep 28 14:38:32 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606312, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:32.456012 20358 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:38:32.456980 20358 webserver.cc:460] Webserver started at http://127.6.23.195:38567/ using document root <none> and password file <none>
I20260928 14:38:32.457579 20358 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:32.457675 20358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:32.457926 20358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:32.460209 20358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "310344369ce749cda4cad0f178cca760"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "fac4eca29fb96808f75c67b0c2987e61"
server_key_iv: "a437456ad70459b7799a1e67f6b3e009"
server_key_version: "encryptionkey@0"
I20260928 14:38:32.460836 20358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "310344369ce749cda4cad0f178cca760"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "fac4eca29fb96808f75c67b0c2987e61"
server_key_iv: "a437456ad70459b7799a1e67f6b3e009"
server_key_version: "encryptionkey@0"
I20260928 14:38:32.464916 20358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:38:32.467491 20373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:32.468814 20358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:38:32.468988 20358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "310344369ce749cda4cad0f178cca760"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "fac4eca29fb96808f75c67b0c2987e61"
server_key_iv: "a437456ad70459b7799a1e67f6b3e009"
server_key_version: "encryptionkey@0"
I20260928 14:38:32.469113 20358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:32.483693 20358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:32.486685 20358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:32.486902 20358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:32.487609 20358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:32.488449 20358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:32.488518 20358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:32.488587 20358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:32.488633 20358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:32.497900 20358 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:33375
I20260928 14:38:32.497936 20486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:33375 every 8 connection(s)
I20260928 14:38:32.499027 20358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:38:32.507994 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 20358
I20260928 14:38:32.508136 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:38:32.508455 6239 external_mini_cluster.cc:1428] Setting key d0eec688b5934222dd764d9ae8b2544b
Sep 28 14:38:32 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606312, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:32.513887 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.501100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:45007 (local address 127.6.23.254:38565)
0928 14:38:32.501371 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.501376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.502662 (+ 1286us) server_negotiation.cc:407] Connection header received
0928 14:38:32.503577 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.503580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.503631 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.503699 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:32.505062 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.505636 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.506923 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.507155 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.507326 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.510126 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:32.510129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:32.510131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:32.510157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:32.511734 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:32.512280 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:32.512284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:32.512286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:32.512369 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:32.512727 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:32.512731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:32.512733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:32.512965 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:32.513072 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.513595 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.513725 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 14:38:32.514777 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.501548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:45007)
0928 14:38:32.502496 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.502541 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:32.503347 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:32.503919 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:32.503927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:32.504343 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:32.504902 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.504917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.505794 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.505797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.506779 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.506789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.507434 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.507438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.507613 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.508343 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:32.508360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:32.509940 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:32.511868 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:32.511875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:32.511908 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:32.512170 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:32.512474 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:32.512477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:32.512479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:32.512627 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:32.513071 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:32.513079 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:32.513350 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 14:38:32.513556 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 14:38:32.513918 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":558,"threads_started":1}
I20260928 14:38:32.516161 20487 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38565
I20260928 14:38:32.516417 20487 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:32.516891 20487 heartbeater.cc:511] Master 127.6.23.254:38565 requested a full tablet report, sending...
I20260928 14:38:32.517896 20028 ts_manager.cc:194] Registered new tserver with Master: 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
I20260928 14:38:32.518460 20028 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:45007
I20260928 14:38:32.522526 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:38:32.531685 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.524668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.6.23.254:38565)
0928 14:38:32.524800 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.524804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.524994 (+ 190us) server_negotiation.cc:407] Connection header received
0928 14:38:32.525121 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.525124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.525173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.525265 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:32.525920 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.526432 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.527314 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.527545 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.527625 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.528907 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:32.528910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:32.528912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:32.528935 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:32.530180 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:32.530566 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:32.530569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:32.530570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:32.530645 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:32.530866 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:32.530868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:32.530870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:32.531080 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:32.531182 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.531445 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.531559 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:38:32.534708 20028 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:38:32.537909 20028 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:38:32.553201 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.548299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.6.23.194:37691)
0928 14:38:32.548659 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.548668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.548686 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:38:32.548746 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.548749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.548887 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.549051 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:32.549215 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.549993 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.551606 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.552465 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.552578 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.552727 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.552918 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.553037 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260928 14:38:32.553191 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.547904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37691 (local address 127.0.0.1:51654)
0928 14:38:32.548344 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.548357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:32.548430 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:32.549023 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:32.549026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:32.549043 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:32.549133 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.549139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.550133 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.550136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.551489 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.551499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.552595 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.552599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.552722 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.552767 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:38:32.552883 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 14:38:32.553057 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":114,"thread_start_us":89,"threads_started":1}
I20260928 14:38:32.553226 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.547605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:33933 (local address 127.0.0.1:47228)
0928 14:38:32.548311 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.548325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:32.548427 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:32.549119 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:32.549122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:32.549138 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:32.549230 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.549236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.550684 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.550687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.551858 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.551865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.552710 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.552713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.552783 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.552799 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 14:38:32.552857 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 14:38:32.553054 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":112,"thread_start_us":95,"threads_started":1}
I20260928 14:38:32.553514 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.546746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:33375 (local address 127.0.0.1:42280)
0928 14:38:32.547205 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.547329 (+ 124us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:32.547469 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:32.548308 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:32.548315 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:32.548337 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:32.548492 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.548502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.549889 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.549893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.551573 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.551585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.552959 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.552962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.553047 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.553067 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:38:32.553134 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 14:38:32.553192 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":100,"thread_start_us":122,"threads_started":1}
I20260928 14:38:32.553920 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.546878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.6.23.195:33375)
0928 14:38:32.547213 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.547252 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.547429 (+ 177us) server_negotiation.cc:407] Connection header received
0928 14:38:32.547680 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.547690 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.547934 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.548101 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:32.548620 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.549753 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.551681 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.552465 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.552578 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.552727 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.553264 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.553785 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":157,"threads_started":1}
I20260928 14:38:32.553922 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.547927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.6.23.193:33933)
0928 14:38:32.548647 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.548663 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.548687 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:38:32.548760 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.548765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.548965 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.549383 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:32.549525 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.550538 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.551981 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.552580 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.553289 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.553601 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.553655 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.553774 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":114,"threads_started":1}
I20260928 14:38:32.556841 20421 tablet_service.cc:1467] Processing CreateTablet for tablet d4a4e6c468c841219557106bda3b4883 (DEFAULT_TABLE table=test-table [id=3e0ac92621724bb4b046f5ca9897f93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:32.557000 20287 tablet_service.cc:1467] Processing CreateTablet for tablet d4a4e6c468c841219557106bda3b4883 (DEFAULT_TABLE table=test-table [id=3e0ac92621724bb4b046f5ca9897f93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:32.557199 20153 tablet_service.cc:1467] Processing CreateTablet for tablet d4a4e6c468c841219557106bda3b4883 (DEFAULT_TABLE table=test-table [id=3e0ac92621724bb4b046f5ca9897f93f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:32.557920 20421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a4e6c468c841219557106bda3b4883. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:32.558051 20153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a4e6c468c841219557106bda3b4883. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:32.558051 20287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4a4e6c468c841219557106bda3b4883. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:32.565263 20504 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Bootstrap starting.
I20260928 14:38:32.565891 20506 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Bootstrap starting.
I20260928 14:38:32.565891 20505 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Bootstrap starting.
I20260928 14:38:32.567451 20504 tablet_bootstrap.cc:654] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:32.568161 20505 tablet_bootstrap.cc:654] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:32.568579 20506 tablet_bootstrap.cc:654] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:32.568580 20504 log.cc:824] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:32.569082 20505 log.cc:824] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:32.569358 20506 log.cc:824] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:32.570999 20504 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: No bootstrap required, opened a new log
I20260928 14:38:32.571077 20505 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: No bootstrap required, opened a new log
I20260928 14:38:32.571218 20506 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: No bootstrap required, opened a new log
I20260928 14:38:32.571205 20504 ts_tablet_manager.cc:1397] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 14:38:32.571343 20505 ts_tablet_manager.cc:1397] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:38:32.571419 20506 ts_tablet_manager.cc:1397] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 14:38:32.575202 20505 raft_consensus.cc:348] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.575539 20505 raft_consensus.cc:374] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:32.575613 20505 raft_consensus.cc:729] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585b9c9bfbb1447ea05b2db1f1bfbd2c, State: Initialized, Role: FOLLOWER
I20260928 14:38:32.575448 20506 raft_consensus.cc:348] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.575448 20504 raft_consensus.cc:348] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.575721 20504 raft_consensus.cc:374] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:32.575749 20506 raft_consensus.cc:374] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:32.575802 20506 raft_consensus.cc:729] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabcee39ccd945ef92f4c9edd826d45c, State: Initialized, Role: FOLLOWER
I20260928 14:38:32.575816 20504 raft_consensus.cc:729] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310344369ce749cda4cad0f178cca760, State: Initialized, Role: FOLLOWER
I20260928 14:38:32.576160 20505 consensus_queue.cc:260] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.576248 20504 consensus_queue.cc:260] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.576248 20506 consensus_queue.cc:260] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.577242 20504 ts_tablet_manager.cc:1428] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 14:38:32.577242 20506 ts_tablet_manager.cc:1428] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 14:38:32.577248 20505 ts_tablet_manager.cc:1428] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260928 14:38:32.577594 20219 heartbeater.cc:503] Master 127.6.23.254:38565 was elected leader, sending a full tablet report...
I20260928 14:38:32.577610 20487 heartbeater.cc:503] Master 127.6.23.254:38565 was elected leader, sending a full tablet report...
I20260928 14:38:32.577646 20353 heartbeater.cc:503] Master 127.6.23.254:38565 was elected leader, sending a full tablet report...
W20260928 14:38:32.750113 20488 tablet.cc:2369] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:38:32.775467 20354 tablet.cc:2369] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:38:32.795918 20220 tablet.cc:2369] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:38:32.873003 20511 raft_consensus.cc:482] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:32.873248 20511 raft_consensus.cc:504] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.874549 20511 leader_election.cc:288] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 585b9c9bfbb1447ea05b2db1f1bfbd2c (127.6.23.193:33933), dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691)
I20260928 14:38:32.878266 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.874933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:52483 (local address 127.6.23.193:33933)
0928 14:38:32.875090 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.875096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.875114 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:38:32.875281 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.875286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.875363 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.875474 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:32.875736 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.876250 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.877288 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.877697 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.877830 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.877887 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.878062 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.878139 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:38:32.878266 20489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.874850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:33933 (local address 127.6.23.195:52483)
0928 14:38:32.875001 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.875015 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:32.875130 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:32.875492 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:32.875494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:32.875514 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:32.875632 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.875638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.876359 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.876363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.877168 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.877176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.877820 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.877824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.877936 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.877960 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:38:32.878046 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:38:32.878123 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":22}
I20260928 14:38:32.878702 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.875157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37691 (local address 127.6.23.195:48799)
0928 14:38:32.875600 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.875610 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:32.875722 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:32.875961 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:32.875964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:32.875980 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:32.876105 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.876110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.876805 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.876811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.877613 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:32.877622 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.878264 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.878268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:32.878375 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.878400 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:38:32.878491 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:38:32.878563 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":1,"thread_start_us":155,"threads_started":1}
I20260928 14:38:32.878739 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.875326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:48799 (local address 127.6.23.194:37691)
0928 14:38:32.875463 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.875467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.875692 (+ 225us) server_negotiation.cc:407] Connection header received
0928 14:38:32.875820 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.875822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.875864 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.875943 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:32.876214 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.876694 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.877741 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.878141 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.878272 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.878322 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.878522 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.878604 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:38:32.879034 20173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a4e6c468c841219557106bda3b4883" candidate_uuid: "310344369ce749cda4cad0f178cca760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" is_pre_election: true
I20260928 14:38:32.879390 20173 raft_consensus.cc:2454] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310344369ce749cda4cad0f178cca760 in term 0.
I20260928 14:38:32.879909 20375 leader_election.cc:302] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310344369ce749cda4cad0f178cca760, 585b9c9bfbb1447ea05b2db1f1bfbd2c; no voters:
I20260928 14:38:32.880180 20307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a4e6c468c841219557106bda3b4883" candidate_uuid: "310344369ce749cda4cad0f178cca760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabcee39ccd945ef92f4c9edd826d45c" is_pre_election: true
I20260928 14:38:32.880223 20511 raft_consensus.cc:2781] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:38:32.880322 20511 raft_consensus.cc:482] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:38:32.880367 20511 raft_consensus.cc:3037] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:32.880537 20307 raft_consensus.cc:2454] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310344369ce749cda4cad0f178cca760 in term 0.
I20260928 14:38:32.881541 20511 raft_consensus.cc:504] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.881850 20511 leader_election.cc:288] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [CANDIDATE]: Term 1 election: Requested vote from peers 585b9c9bfbb1447ea05b2db1f1bfbd2c (127.6.23.193:33933), dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691)
I20260928 14:38:32.882287 20173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a4e6c468c841219557106bda3b4883" candidate_uuid: "310344369ce749cda4cad0f178cca760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c"
I20260928 14:38:32.882329 20307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4a4e6c468c841219557106bda3b4883" candidate_uuid: "310344369ce749cda4cad0f178cca760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabcee39ccd945ef92f4c9edd826d45c"
I20260928 14:38:32.882440 20173 raft_consensus.cc:3037] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:32.882479 20307 raft_consensus.cc:3037] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:32.883589 20307 raft_consensus.cc:2454] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310344369ce749cda4cad0f178cca760 in term 1.
I20260928 14:38:32.883628 20173 raft_consensus.cc:2454] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310344369ce749cda4cad0f178cca760 in term 1.
I20260928 14:38:32.884006 20375 leader_election.cc:302] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310344369ce749cda4cad0f178cca760, 585b9c9bfbb1447ea05b2db1f1bfbd2c; no voters:
I20260928 14:38:32.884248 20511 raft_consensus.cc:2781] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:32.884536 20511 raft_consensus.cc:686] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 1 LEADER]: Becoming Leader. State: Replica: 310344369ce749cda4cad0f178cca760, State: Running, Role: LEADER
I20260928 14:38:32.884894 20511 consensus_queue.cc:237] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:32.888166 20028 catalog_manager.cc:5495] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 reported cstate change: term changed from 0 to 1, leader changed from <none> to 310344369ce749cda4cad0f178cca760 (127.6.23.195). New cstate: current_term: 1 leader_uuid: "310344369ce749cda4cad0f178cca760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } health_report { overall_health: UNKNOWN } } }
I20260928 14:38:32.918672 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.915192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.6.23.195:33375)
0928 14:38:32.915352 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:32.915355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:32.915369 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:38:32.915505 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:32.915508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:32.915566 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:32.915659 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:38:32.915882 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.916474 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.917307 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:32.917526 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:32.917601 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:32.917875 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:38:32.918309 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:38:32.918403 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:32.918479 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 14:38:32.918534 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:38:32.925002 20307 raft_consensus.cc:1261] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 1 FOLLOWER]: Refusing update from remote peer 310344369ce749cda4cad0f178cca760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:32.925012 20173 raft_consensus.cc:1261] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Refusing update from remote peer 310344369ce749cda4cad0f178cca760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:32.925765 20511 consensus_queue.cc:1035] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:32.925988 20514 consensus_queue.cc:1035] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:32.933122 20517 mvcc.cc:205] Tried to move back new op lower bound from 7334323457732517888 to 7334323457579184128. Current Snapshot: MvccSnapshot[applied={T|T < 7334323457732517888}]
I20260928 14:38:32.933730 20519 mvcc.cc:205] Tried to move back new op lower bound from 7334323457732517888 to 7334323457579184128. Current Snapshot: MvccSnapshot[applied={T|T < 7334323457732517888}]
I20260928 14:38:32.935371 20518 mvcc.cc:205] Tried to move back new op lower bound from 7334323457732517888 to 7334323457579184128. Current Snapshot: MvccSnapshot[applied={T|T < 7334323457732517888}]
I20260928 14:38:32.942997 20028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:38:32.943185 20028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:38:32.945854 20028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Sending DeleteTablet for 3 replicas of tablet d4a4e6c468c841219557106bda3b4883
I20260928 14:38:32.946928 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 20090
I20260928 14:38:32.946844 20421 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a4e6c468c841219557106bda3b4883 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:32 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:42280
I20260928 14:38:32.947019 20153 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a4e6c468c841219557106bda3b4883 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:32 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:47228
I20260928 14:38:32.948197 20287 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a4e6c468c841219557106bda3b4883 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:32 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:51654
I20260928 14:38:32.948784 20528 tablet_replica.cc:330] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: stopping tablet replica
I20260928 14:38:32.949208 20528 raft_consensus.cc:2229] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:38:32.949669 20528 raft_consensus.cc:2258] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:32.951073 20527 tablet_replica.cc:330] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: stopping tablet replica
I20260928 14:38:32.951440 20527 raft_consensus.cc:2229] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:38:32.951979 20527 raft_consensus.cc:2258] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:32.952385 20528 ts_tablet_manager.cc:1905] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:32.955327 20528 ts_tablet_manager.cc:1918] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:32.955511 20528 log.cc:1196] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/d4a4e6c468c841219557106bda3b4883
I20260928 14:38:32.956362 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 20224
W20260928 14:38:32.956408 20013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:33933 (error 108)
I20260928 14:38:32.957724 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.957349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:33933 (local address 127.0.0.1:47236)
0928 14:38:32.957508 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.957609 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:33933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:38:32.958441 20013 catalog_manager.cc:4584] TS 585b9c9bfbb1447ea05b2db1f1bfbd2c (127.6.23.193:33933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4a4e6c468c841219557106bda3b4883: Network error: Client connection negotiation failed: client connection to 127.6.23.193:33933: connect: Connection refused (error 111)
I20260928 14:38:32.958922 20527 ts_tablet_manager.cc:1905] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:32.961385 20527 ts_tablet_manager.cc:1918] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:32.961510 20527 log.cc:1196] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/d4a4e6c468c841219557106bda3b4883
I20260928 14:38:32.961872 20527 ts_tablet_manager.cc:1939] T d4a4e6c468c841219557106bda3b4883 P 310344369ce749cda4cad0f178cca760: Deleting consensus metadata
I20260928 14:38:32.963122 20013 catalog_manager.cc:4841] TS 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375): tablet d4a4e6c468c841219557106bda3b4883 (table test-table [id=3e0ac92621724bb4b046f5ca9897f93f]) successfully deleted
I20260928 14:38:32.964066 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 20358
I20260928 14:38:32.965106 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:32.964176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37691 (local address 127.0.0.1:51668)
0928 14:38:32.964925 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:32.964996 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.194:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643}
I20260928 14:38:32.970398 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 19996
I20260928 14:38:32.977275 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:38565
--webserver_interface=127.6.23.254
--webserver_port=37865
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:38565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:33.091687 20530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:33.091982 20530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:33.092025 20530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:33.096189 20530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:33.096261 20530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:33.096282 20530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:33.096298 20530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:33.096313 20530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:33.101769 20530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:38565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:38565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=37865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:33.102990 20530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:33.104176 20530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:33.117796 20537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:33.117758 20536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:33.117762 20539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:33.118148 20530 server_base.cc:1034] running on GCE node
I20260928 14:38:33.118789 20530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:33.120025 20530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:33.121187 20530 hybrid_clock.cc:648] HybridClock initialized: now 1790606313121147 us; error 55 us; skew 500 ppm
Sep 28 14:38:33 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:33.124249 20530 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:38:33.125324 20530 webserver.cc:460] Webserver started at http://127.6.23.254:37865/ using document root <none> and password file <none>
I20260928 14:38:33.125962 20530 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:33.126086 20530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:33.130349 20530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:38:33.132812 20546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:33.134215 20530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:38:33.134374 20530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "3a10326e298041d397dc104bff976c7a"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "d3678809c8b0d8fb0a57434a1d77ee22"
server_key_iv: "924c87b2efa8adabc93d3f1bd6514d64"
server_key_version: "encryptionkey@0"
I20260928 14:38:33.135025 20530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:33.146102 20530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:33.149595 20530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:33.149813 20530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:33.157495 20530 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:38565
I20260928 14:38:33.157789 20598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:38565 every 8 connection(s)
I20260928 14:38:33.158754 20530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:38:33.163841 20599 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:38:33.163961 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 20530
I20260928 14:38:33.164474 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:33933
--local_ip_for_outbound_sockets=127.6.23.193
--tserver_master_addrs=127.6.23.254:38565
--webserver_port=38933
--webserver_interface=127.6.23.193
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:38:33.166438 20599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Bootstrap starting.
I20260928 14:38:33.185407 20599 log.cc:824] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:33.194367 20599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:38:33.194782 20599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Bootstrap complete.
I20260928 14:38:33.198962 20599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:33.199472 20599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a10326e298041d397dc104bff976c7a, State: Initialized, Role: FOLLOWER
I20260928 14:38:33.200001 20599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:33.200121 20599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:38:33.200217 20599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:38:33.200347 20599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:38:33.202411 20599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:33.202778 20599 leader_election.cc:302] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a10326e298041d397dc104bff976c7a; no voters:
I20260928 14:38:33.203130 20599 leader_election.cc:288] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:38:33.203222 20603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:38:33.203452 20603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 2 LEADER]: Becoming Leader. State: Replica: 3a10326e298041d397dc104bff976c7a, State: Running, Role: LEADER
I20260928 14:38:33.203778 20603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:33.204517 20599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:33.204842 20605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3a10326e298041d397dc104bff976c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } } }
I20260928 14:38:33.204979 20605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:33.205214 20604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a10326e298041d397dc104bff976c7a. Latest consensus state: current_term: 2 leader_uuid: "3a10326e298041d397dc104bff976c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } } }
I20260928 14:38:33.205317 20604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:33.205345 20607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:38:33.210584 20607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:38:33.211421 20607 catalog_manager.cc:1259] Loaded cluster ID: ba768ef0a3694d9ab92ace02dbb61fd3
I20260928 14:38:33.211521 20607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:38:33.212666 20607 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 14:38:33.215663 20607 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 14:38:33.223886 20607 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:38:33.224834 20607 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Loaded TSK: 0
I20260928 14:38:33.226141 20607 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:38:33.291658 20601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:33.292003 20601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:33.292089 20601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:33.296535 20601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:33.296640 20601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:33.296741 20601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:38:33.302412 20601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:33933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=38933
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:33.303727 20601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:33.304795 20601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:33.315296 20626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:33.315418 20627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:33.315296 20629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:33.316073 20601 server_base.cc:1034] running on GCE node
I20260928 14:38:33.316501 20601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:33.317224 20601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:33.318413 20601 hybrid_clock.cc:648] HybridClock initialized: now 1790606313318383 us; error 47 us; skew 500 ppm
Sep 28 14:38:33 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:33.321596 20601 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:38:33.322600 20601 webserver.cc:460] Webserver started at http://127.6.23.193:38933/ using document root <none> and password file <none>
I20260928 14:38:33.323216 20601 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:33.323393 20601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:33.327790 20601 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 14:38:33.330260 20636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:33.331756 20601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:38:33.331907 20601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "18a24d3950cac92c09860790b7a88704"
server_key_iv: "d5069ee94f209acb5c004b93b19ccfbd"
server_key_version: "encryptionkey@0"
I20260928 14:38:33.332504 20601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:33.353288 20601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:33.357167 20601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:33.357409 20601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:33.358072 20601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:33.359350 20643 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:38:33.363855 20601 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:38:33.363933 20601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 14:38:33.364013 20601 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:38:33.365772 20601 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:38:33.365836 20601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 14:38:33.366258 20643 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Bootstrap starting.
I20260928 14:38:33.377226 20601 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:33933
I20260928 14:38:33.377228 20750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:33933 every 8 connection(s)
I20260928 14:38:33.378728 20601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:38:33.382195 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 20601
I20260928 14:38:33.382695 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:37691
--local_ip_for_outbound_sockets=127.6.23.194
--tserver_master_addrs=127.6.23.254:38565
--webserver_port=40129
--webserver_interface=127.6.23.194
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:38:33.388877 20643 log.cc:824] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:38:33 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:33.397367 20643 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:38:33.397775 20643 tablet_bootstrap.cc:492] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Bootstrap complete.
I20260928 14:38:33.399160 20643 ts_tablet_manager.cc:1397] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Time spent bootstrapping tablet: real 0.033s user 0.021s sys 0.008s
I20260928 14:38:33.400090 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.381074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:35545 (local address 127.6.23.254:38565)
0928 14:38:33.381452 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:33.381473 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:33.381822 (+ 349us) server_negotiation.cc:407] Connection header received
0928 14:38:33.383500 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:33.383523 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:33.384146 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:33.384606 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:33.386013 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.387897 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.389540 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.390096 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.390320 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.393614 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:33.393624 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:33.393646 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:33.393688 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:33.396089 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.396735 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.396742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.396747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.396913 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.397492 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.397495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.397497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.398021 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:33.398144 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:33.398839 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 14:38:33.399161 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":124,"threads_started":1}
I20260928 14:38:33.400100 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.381114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:35545)
0928 14:38:33.381645 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:33.381688 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:33.382917 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:33.384687 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:33.384697 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:33.385268 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:33.385804 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.385820 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.388074 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.388078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.389402 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.389412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.390264 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.390270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.390565 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.391350 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:33.391376 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:33.393383 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:33.396233 (+ 2850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:33.396240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:33.396254 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:33.396610 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:33.397067 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:33.397071 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:33.397073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:33.397245 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:33.398161 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:33.398170 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:33.398498 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 14:38:33.398702 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 14:38:33.399076 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":153,"threads_started":1}
I20260928 14:38:33.402199 20643 raft_consensus.cc:348] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:33.402745 20643 raft_consensus.cc:729] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585b9c9bfbb1447ea05b2db1f1bfbd2c, State: Initialized, Role: FOLLOWER
I20260928 14:38:33.403198 20751 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38565
I20260928 14:38:33.403293 20643 consensus_queue.cc:260] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } } peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } }
I20260928 14:38:33.403468 20751 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:33.404043 20751 heartbeater.cc:511] Master 127.6.23.254:38565 requested a full tablet report, sending...
I20260928 14:38:33.404302 20643 ts_tablet_manager.cc:1428] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:38:33.406723 20563 ts_manager.cc:194] Registered new tserver with Master: 585b9c9bfbb1447ea05b2db1f1bfbd2c (127.6.23.193:33933)
I20260928 14:38:33.410228 20563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:35545
I20260928 14:38:33.412883 20751 heartbeater.cc:503] Master 127.6.23.254:38565 was elected leader, sending a full tablet report...
Sep 28 14:38:33 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:38:33.420599 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.409668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:33933 (local address 127.0.0.1:47238)
0928 14:38:33.410082 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:33.410130 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:33.410264 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:33.411024 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:33.411031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:33.411484 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:33.411600 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.411607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.412562 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.412567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.413321 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.413332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.414079 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.414085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.414202 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.414808 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:33.414822 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:33.416420 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:33.418645 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:33.418654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:33.418658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:33.419051 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:33.419415 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:33.419418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:33.419420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:33.419503 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:33.420045 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:33.420050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:33.420166 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 14:38:33.420290 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:38:33.420460 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
I20260928 14:38:33.420957 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.409777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.6.23.193:33933)
0928 14:38:33.410546 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:33.410558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:33.410582 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:38:33.410657 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:33.410662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:33.410866 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:33.411021 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:33.411726 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.412441 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.413448 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.413930 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.414049 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.416565 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:33.416575 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:33.416588 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:33.416614 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:33.418524 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.419181 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.419188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.419190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.419303 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.419590 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.419593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.419595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.419899 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:33.420040 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:33.420389 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 14:38:33.420564 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":94,"threads_started":1}
I20260928 14:38:33.421945 20685 tablet_service.cc:1514] Processing DeleteTablet for tablet d4a4e6c468c841219557106bda3b4883 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:32 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:47238
I20260928 14:38:33.422847 20760 tablet_replica.cc:330] stopping tablet replica
I20260928 14:38:33.423173 20760 raft_consensus.cc:2229] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:38:33.423460 20760 raft_consensus.cc:2258] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:33.424481 20760 ts_tablet_manager.cc:1905] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:33.428117 20760 ts_tablet_manager.cc:1918] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:33.428200 20760 log.cc:1196] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/d4a4e6c468c841219557106bda3b4883
I20260928 14:38:33.428504 20760 ts_tablet_manager.cc:1939] T d4a4e6c468c841219557106bda3b4883 P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Deleting consensus metadata
I20260928 14:38:33.430286 20548 catalog_manager.cc:4841] TS 585b9c9bfbb1447ea05b2db1f1bfbd2c (127.6.23.193:33933): tablet d4a4e6c468c841219557106bda3b4883 (table test-table [id=3e0ac92621724bb4b046f5ca9897f93f]) successfully deleted
W20260928 14:38:33.508373 20755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:33.508703 20755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:33.508754 20755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:33.513079 20755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:33.513155 20755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:33.513218 20755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:38:33.518939 20755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:37691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=40129
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:33.520224 20755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:33.521267 20755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:33.531698 20768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:33.531703 20766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:33.531729 20765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:33.531926 20755 server_base.cc:1034] running on GCE node
I20260928 14:38:33.532356 20755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:33.533138 20755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:33.534359 20755 hybrid_clock.cc:648] HybridClock initialized: now 1790606313534342 us; error 27 us; skew 500 ppm
Sep 28 14:38:33 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:33.537356 20755 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:38:33.538384 20755 webserver.cc:460] Webserver started at http://127.6.23.194:40129/ using document root <none> and password file <none>
I20260928 14:38:33.539111 20755 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:33.539222 20755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:33.543485 20755 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:38:33.545831 20775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:33.547385 20755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:38:33.547533 20755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "dabcee39ccd945ef92f4c9edd826d45c"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "87b6e196c9486f8d682742219f7c45a8"
server_key_iv: "9030694fdc42236cc4307bf8f04cd202"
server_key_version: "encryptionkey@0"
I20260928 14:38:33.548158 20755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:33.560190 20755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:33.563627 20755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:33.563849 20755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:33.564492 20755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:33.565668 20782 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 14:38:33.569643 20782 ts_tablet_manager.cc:1852] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 14:38:33.570047 20782 ts_tablet_manager.cc:1905] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:33.574172 20782 ts_tablet_manager.cc:1918] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:33.574331 20782 ts_tablet_manager.cc:1939] T d4a4e6c468c841219557106bda3b4883 P dabcee39ccd945ef92f4c9edd826d45c: Deleting consensus metadata
I20260928 14:38:33.574669 20755 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 14:38:33.574749 20755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260928 14:38:33.574869 20755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:33.574914 20755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:33.584213 20755 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:37691
I20260928 14:38:33.584254 20889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:37691 every 8 connection(s)
I20260928 14:38:33.585146 20755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:38:33.590839 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 20755
I20260928 14:38:33.591351 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:33375
--local_ip_for_outbound_sockets=127.6.23.195
--tserver_master_addrs=127.6.23.254:38565
--webserver_port=38567
--webserver_interface=127.6.23.195
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:38:33 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:33.599452 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.587001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:54991 (local address 127.6.23.254:38565)
0928 14:38:33.587159 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:33.587163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:33.588134 (+ 971us) server_negotiation.cc:407] Connection header received
0928 14:38:33.589032 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:33.589035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:33.589088 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:33.589164 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:33.590292 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.590994 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.592261 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.592691 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.592788 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.595781 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:33.595784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:33.595786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:33.595811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:33.597321 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.597851 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.597859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.597860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.597931 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.598295 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.598298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.598299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.598496 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:33.598605 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:33.599102 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 14:38:33.599271 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:38:33.600265 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.587523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:54991)
0928 14:38:33.587978 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:33.588023 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:33.588836 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:33.589328 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:33.589335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:33.589701 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:33.590169 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.590182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.591114 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.591117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.592139 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.592148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.592864 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.592870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.593099 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.593755 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:33.593783 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:33.595593 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:33.597447 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:33.597453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:33.597468 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:33.597734 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:33.598029 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:33.598032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:33.598034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:33.598188 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:33.598603 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:33.598611 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:33.598859 (+ 248us) client_negotiation.cc:769] Sending connection context
0928 14:38:33.599048 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 14:38:33.599410 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260928 14:38:33.601701 20890 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38565
I20260928 14:38:33.601958 20890 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:33.602463 20890 heartbeater.cc:511] Master 127.6.23.254:38565 requested a full tablet report, sending...
I20260928 14:38:33.603648 20563 ts_manager.cc:194] Registered new tserver with Master: dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691)
I20260928 14:38:33.604416 20563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:54991
W20260928 14:38:33.709093 20893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:33.709362 20893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:33.709406 20893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:33.713441 20893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:33.713503 20893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:33.713562 20893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:38:33.719543 20893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:33375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=38567
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:33.720613 20893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:33.721621 20893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:33.732074 20899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:33.732057 20901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:33.732057 20898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:33.732431 20893 server_base.cc:1034] running on GCE node
I20260928 14:38:33.732895 20893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:33.733629 20893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:33.734789 20893 hybrid_clock.cc:648] HybridClock initialized: now 1790606313734768 us; error 36 us; skew 500 ppm
Sep 28 14:38:33 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:33.737784 20893 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:38:33.738787 20893 webserver.cc:460] Webserver started at http://127.6.23.195:38567/ using document root <none> and password file <none>
I20260928 14:38:33.739504 20893 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:33.739584 20893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:33.743438 20893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:38:33.745383 20908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:33.746623 20893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:38:33.746762 20893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "310344369ce749cda4cad0f178cca760"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "fac4eca29fb96808f75c67b0c2987e61"
server_key_iv: "a437456ad70459b7799a1e67f6b3e009"
server_key_version: "encryptionkey@0"
I20260928 14:38:33.747380 20893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:33.756738 20893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:33.760143 20893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:33.760345 20893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:33.760947 20893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:33.761794 20893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:33.761895 20893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:33.761956 20893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:33.761996 20893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:33.770959 20893 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:33375
I20260928 14:38:33.770980 21021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:33375 every 8 connection(s)
I20260928 14:38:33.772059 20893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:38:33.778839 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 20893
Sep 28 14:38:33 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:33.786706 20753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.773880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:54487 (local address 127.6.23.254:38565)
0928 14:38:33.774031 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:33.774035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:33.775082 (+ 1047us) server_negotiation.cc:407] Connection header received
0928 14:38:33.776011 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:33.776014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:33.776066 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:33.776156 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:33.777362 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.777910 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.779450 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.779797 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.779917 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.782502 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:33.782506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:33.782508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:33.782538 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:33.784254 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.784935 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.784945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.784947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.785055 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.785490 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.785495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.785497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.785676 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:33.785751 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:33.786418 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 14:38:33.786541 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:38:33.787832 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.774294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:54487)
0928 14:38:33.774927 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:33.774967 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:33.775791 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:33.776351 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:33.776358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:33.776714 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:33.777205 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.777219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.778046 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.778049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.779313 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.779324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.779978 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.779982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.780167 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.780827 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:33.780845 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:33.782327 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:33.784408 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:33.784421 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:33.784439 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:33.784805 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:33.785161 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:33.785165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:33.785168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:33.785354 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:33.785784 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:33.785791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:33.786141 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 14:38:33.786357 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 14:38:33.786692 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":111,"threads_started":1}
I20260928 14:38:33.787977 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.779627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.6.23.254:38565)
0928 14:38:33.779927 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:33.779932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:33.780033 (+ 101us) server_negotiation.cc:407] Connection header received
0928 14:38:33.780205 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:33.780208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:33.780257 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:33.780346 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:33.781077 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.781599 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.782528 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.782780 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.782888 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.784429 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:33.784433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:33.784436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:33.784461 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:33.786143 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.786626 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.786633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.786637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.786724 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:33.787041 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:33.787045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:33.787046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:33.787208 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:33.787423 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:33.787656 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 14:38:33.787764 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":107,"threads_started":1}
I20260928 14:38:33.789188 21022 heartbeater.cc:347] Connected to a master server at 127.6.23.254:38565
I20260928 14:38:33.789458 21022 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:33.789983 21022 heartbeater.cc:511] Master 127.6.23.254:38565 requested a full tablet report, sending...
I20260928 14:38:33.791069 20563 ts_manager.cc:194] Registered new tserver with Master: 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
I20260928 14:38:33.791690 20563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:54487
I20260928 14:38:33.801571 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:38:33.806054 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.802190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42868 (local address 127.6.23.254:38565)
0928 14:38:33.802318 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:33.802329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:33.802559 (+ 230us) server_negotiation.cc:407] Connection header received
0928 14:38:33.802688 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:33.802690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:33.802789 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:33.802877 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:38:33.803068 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.803664 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.804580 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.804811 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.804894 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.805146 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:38:33.805612 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:38:33.805705 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:33.805882 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:38:33.805948 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:38:33.806682 20563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:38:33.810034 20563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:38:33.818336 20685 tablet_service.cc:1467] Processing CreateTablet for tablet 03add453fca64c5b910f53d63b0f57db (DEFAULT_TABLE table=test-table [id=1b38d12fe38d4c01a867615eb2e122dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:33.819046 20685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03add453fca64c5b910f53d63b0f57db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:33.821907 20643 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Bootstrap starting.
I20260928 14:38:33.822803 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.817063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37691 (local address 127.0.0.1:51684)
0928 14:38:33.817209 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:33.817227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:33.817324 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:33.818269 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:33.818273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:33.818291 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:33.818402 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.818412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.819936 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.819940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.821152 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.821163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.822328 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.822334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.822440 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.822490 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 14:38:33.822568 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:38:33.822651 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:38:33.822861 20643 tablet_bootstrap.cc:654] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:33.822902 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.817290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.6.23.194:37691)
0928 14:38:33.817733 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:33.817746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:33.817768 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:38:33.817860 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:33.817866 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:33.818116 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:33.818282 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:33.818615 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.819781 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.821287 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.822177 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.822328 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.822496 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:33.822618 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:38:33.822716 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":150,"threads_started":1}
I20260928 14:38:33.823891 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.817288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:33375 (local address 127.0.0.1:42292)
0928 14:38:33.818065 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:33.818081 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:33.818180 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:33.818877 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:33.818880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:33.818896 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:33.818986 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.818991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.820304 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.820307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.821847 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:33.821859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.823469 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.823472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:33.823563 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.823585 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:38:33.823680 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:38:33.823750 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":137,"threads_started":1}
I20260928 14:38:33.824621 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:33.817738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.6.23.195:33375)
0928 14:38:33.818067 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:33.818083 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:33.818152 (+ 69us) server_negotiation.cc:407] Connection header received
0928 14:38:33.818334 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:33.818339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:33.818555 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:33.818704 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:33.819126 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.820177 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.822039 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:33.823134 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:33.823256 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:33.823406 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:33.824362 (+ 956us) server_negotiation.cc:299] Negotiation successful
0928 14:38:33.824473 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":145,"threads_started":1}
I20260928 14:38:33.824710 20643 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: No bootstrap required, opened a new log
I20260928 14:38:33.824783 20643 ts_tablet_manager.cc:1397] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:38:33.824592 20824 tablet_service.cc:1467] Processing CreateTablet for tablet 03add453fca64c5b910f53d63b0f57db (DEFAULT_TABLE table=test-table [id=1b38d12fe38d4c01a867615eb2e122dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:33.825237 20643 raft_consensus.cc:348] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:33.825361 20643 raft_consensus.cc:374] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:33.825407 20643 raft_consensus.cc:729] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585b9c9bfbb1447ea05b2db1f1bfbd2c, State: Initialized, Role: FOLLOWER
I20260928 14:38:33.825559 20643 consensus_queue.cc:260] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:33.825794 20824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03add453fca64c5b910f53d63b0f57db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:33.825837 20643 ts_tablet_manager.cc:1428] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:38:33.827754 20956 tablet_service.cc:1467] Processing CreateTablet for tablet 03add453fca64c5b910f53d63b0f57db (DEFAULT_TABLE table=test-table [id=1b38d12fe38d4c01a867615eb2e122dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:33.828852 20956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03add453fca64c5b910f53d63b0f57db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:33.830345 20782 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: Bootstrap starting.
I20260928 14:38:33.832525 20782 tablet_bootstrap.cc:654] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:33.833244 20782 log.cc:824] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:33.835124 20782 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: No bootstrap required, opened a new log
I20260928 14:38:33.835336 20782 ts_tablet_manager.cc:1397] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:38:33.836217 21036 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Bootstrap starting.
I20260928 14:38:33.839031 21036 tablet_bootstrap.cc:654] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:33.839462 20782 raft_consensus.cc:348] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:33.839736 20782 raft_consensus.cc:374] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:33.839800 20782 raft_consensus.cc:729] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabcee39ccd945ef92f4c9edd826d45c, State: Initialized, Role: FOLLOWER
I20260928 14:38:33.839982 21036 log.cc:824] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:33.840283 20782 consensus_queue.cc:260] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:33.840958 20890 heartbeater.cc:503] Master 127.6.23.254:38565 was elected leader, sending a full tablet report...
I20260928 14:38:33.841259 20782 ts_tablet_manager.cc:1428] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:38:33.843330 21036 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: No bootstrap required, opened a new log
I20260928 14:38:33.843541 21036 ts_tablet_manager.cc:1397] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260928 14:38:33.846501 21036 raft_consensus.cc:348] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:33.846726 21036 raft_consensus.cc:374] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:33.846791 21036 raft_consensus.cc:729] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310344369ce749cda4cad0f178cca760, State: Initialized, Role: FOLLOWER
I20260928 14:38:33.847287 21036 consensus_queue.cc:260] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:33.847942 21022 heartbeater.cc:503] Master 127.6.23.254:38565 was elected leader, sending a full tablet report...
I20260928 14:38:33.848268 21036 ts_tablet_manager.cc:1428] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260928 14:38:33.880268 20752 tablet.cc:2369] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:38:34.023042 21023 tablet.cc:2369] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:38:34.035594 20757 raft_consensus.cc:482] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:34.035827 20757 raft_consensus.cc:504] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:34.037060 20757 leader_election.cc:288] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691), 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
I20260928 14:38:34.042531 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.037844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:51783 (local address 127.6.23.195:33375)
0928 14:38:34.037986 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:34.037989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:34.038142 (+ 153us) server_negotiation.cc:407] Connection header received
0928 14:38:34.038333 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:34.038336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:34.038412 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:34.038498 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:34.038668 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.039392 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.041133 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.041751 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.041881 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:34.041936 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:34.042309 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 14:38:34.042383 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:38:34.042524 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.037416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:48899 (local address 127.6.23.194:37691)
0928 14:38:34.037563 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:34.037566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:34.037689 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:38:34.037884 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:34.037887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:34.037950 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:34.038050 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:34.038456 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.039343 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.041133 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.041751 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.041881 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:34.041936 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:34.042309 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 14:38:34.042383 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:38:34.042608 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.037312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37691 (local address 127.6.23.193:48899)
0928 14:38:34.037572 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.037586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:34.037745 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:34.038111 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:34.038116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:34.038141 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:34.038334 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:34.038340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.039484 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.039488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:34.040994 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:34.041006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.041906 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.041910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:34.042107 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:34.042187 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 14:38:34.042276 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:38:34.042429 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":273,"spinlock_wait_cycles":64384,"thread_start_us":103,"threads_started":1}
I20260928 14:38:34.042608 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.037718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:33375 (local address 127.6.23.193:51783)
0928 14:38:34.038073 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.038088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:34.038203 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:34.038508 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:34.038510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:34.038525 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:34.038590 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:34.038594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.039497 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.039500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:34.040998 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:34.041006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.041906 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.041910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:34.042103 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:34.042189 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 14:38:34.042276 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:38:34.042426 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":174,"spinlock_wait_cycles":896,"thread_start_us":101,"threads_started":1}
I20260928 14:38:34.043551 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03add453fca64c5b910f53d63b0f57db" candidate_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310344369ce749cda4cad0f178cca760" is_pre_election: true
I20260928 14:38:34.043589 20844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03add453fca64c5b910f53d63b0f57db" candidate_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabcee39ccd945ef92f4c9edd826d45c" is_pre_election: true
I20260928 14:38:34.043891 20844 raft_consensus.cc:2454] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585b9c9bfbb1447ea05b2db1f1bfbd2c in term 0.
I20260928 14:38:34.043895 20976 raft_consensus.cc:2454] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 585b9c9bfbb1447ea05b2db1f1bfbd2c in term 0.
I20260928 14:38:34.044386 20638 leader_election.cc:302] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310344369ce749cda4cad0f178cca760, 585b9c9bfbb1447ea05b2db1f1bfbd2c; no voters:
I20260928 14:38:34.044687 20757 raft_consensus.cc:2781] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:38:34.044780 20757 raft_consensus.cc:482] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:38:34.044824 20757 raft_consensus.cc:3037] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:34.045907 20757 raft_consensus.cc:504] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:34.046277 20757 leader_election.cc:288] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 1 election: Requested vote from peers dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691), 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
I20260928 14:38:34.046696 20844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03add453fca64c5b910f53d63b0f57db" candidate_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabcee39ccd945ef92f4c9edd826d45c"
I20260928 14:38:34.046706 20976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03add453fca64c5b910f53d63b0f57db" candidate_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "310344369ce749cda4cad0f178cca760"
I20260928 14:38:34.046808 20844 raft_consensus.cc:3037] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:34.046852 20976 raft_consensus.cc:3037] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:34.047972 20844 raft_consensus.cc:2454] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585b9c9bfbb1447ea05b2db1f1bfbd2c in term 1.
I20260928 14:38:34.048058 20976 raft_consensus.cc:2454] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 585b9c9bfbb1447ea05b2db1f1bfbd2c in term 1.
I20260928 14:38:34.048317 20638 leader_election.cc:302] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 585b9c9bfbb1447ea05b2db1f1bfbd2c, dabcee39ccd945ef92f4c9edd826d45c; no voters:
I20260928 14:38:34.048493 20757 raft_consensus.cc:2781] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:34.048698 20757 raft_consensus.cc:686] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 LEADER]: Becoming Leader. State: Replica: 585b9c9bfbb1447ea05b2db1f1bfbd2c, State: Running, Role: LEADER
I20260928 14:38:34.048969 20757 consensus_queue.cc:237] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:34.052291 20562 catalog_manager.cc:5495] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 585b9c9bfbb1447ea05b2db1f1bfbd2c (127.6.23.193). New cstate: current_term: 1 leader_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260928 14:38:34.068572 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.065012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.6.23.193:33933)
0928 14:38:34.065290 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:34.065297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:34.065309 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:38:34.065383 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:34.065385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:34.065438 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:34.065540 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:38:34.065752 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.066294 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.067082 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:34.067380 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:34.067461 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:34.067718 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:38:34.068165 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:38:34.068250 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:34.068312 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:38:34.068361 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":108,"threads_started":1}
I20260928 14:38:34.072530 20844 raft_consensus.cc:1261] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 1 FOLLOWER]: Refusing update from remote peer 585b9c9bfbb1447ea05b2db1f1bfbd2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:34.072548 20976 raft_consensus.cc:1261] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 1 FOLLOWER]: Refusing update from remote peer 585b9c9bfbb1447ea05b2db1f1bfbd2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:34.073254 21042 consensus_queue.cc:1035] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:34.073469 20757 consensus_queue.cc:1035] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:34.090095 20562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:38:34.090289 20562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:38:34.092769 20562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Sending DeleteTablet for 3 replicas of tablet 03add453fca64c5b910f53d63b0f57db
I20260928 14:38:34.093454 20824 tablet_service.cc:1514] Processing DeleteTablet for tablet 03add453fca64c5b910f53d63b0f57db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:34 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:51684
I20260928 14:38:34.093528 20685 tablet_service.cc:1514] Processing DeleteTablet for tablet 03add453fca64c5b910f53d63b0f57db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:34 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:47238
I20260928 14:38:34.093724 20956 tablet_service.cc:1514] Processing DeleteTablet for tablet 03add453fca64c5b910f53d63b0f57db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:34 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:42292
I20260928 14:38:34.094149 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 20601
I20260928 14:38:34.094192 21057 tablet_replica.cc:330] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: stopping tablet replica
I20260928 14:38:34.094390 21055 tablet_replica.cc:330] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: stopping tablet replica
I20260928 14:38:34.094543 21055 raft_consensus.cc:2229] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:38:34.095129 21057 raft_consensus.cc:2229] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:38:34.095481 21057 raft_consensus.cc:2258] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:34.095875 21056 tablet_replica.cc:330] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: stopping tablet replica
I20260928 14:38:34.096217 21056 raft_consensus.cc:2229] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:38:34.096511 21056 raft_consensus.cc:2258] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:34.097009 21057 ts_tablet_manager.cc:1905] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:34.097913 21056 ts_tablet_manager.cc:1905] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:34.099267 21057 ts_tablet_manager.cc:1918] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:34.099398 21057 log.cc:1196] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/03add453fca64c5b910f53d63b0f57db
I20260928 14:38:34.099741 21057 ts_tablet_manager.cc:1939] T 03add453fca64c5b910f53d63b0f57db P 310344369ce749cda4cad0f178cca760: Deleting consensus metadata
I20260928 14:38:34.100934 20548 catalog_manager.cc:4841] TS 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375): tablet 03add453fca64c5b910f53d63b0f57db (table test-table [id=1b38d12fe38d4c01a867615eb2e122dd]) successfully deleted
I20260928 14:38:34.101154 21056 ts_tablet_manager.cc:1918] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:34.101260 21056 log.cc:1196] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/03add453fca64c5b910f53d63b0f57db
I20260928 14:38:34.101548 21056 ts_tablet_manager.cc:1939] T 03add453fca64c5b910f53d63b0f57db P dabcee39ccd945ef92f4c9edd826d45c: Deleting consensus metadata
W20260928 14:38:34.101823 20548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:33933 (error 108)
I20260928 14:38:34.102203 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 20755
I20260928 14:38:34.103137 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.102620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:33933 (local address 127.0.0.1:47266)
0928 14:38:34.102951 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.103031 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:33933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
W20260928 14:38:34.103322 20548 catalog_manager.cc:4584] TS 585b9c9bfbb1447ea05b2db1f1bfbd2c (127.6.23.193:33933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03add453fca64c5b910f53d63b0f57db: Network error: Client connection negotiation failed: client connection to 127.6.23.193:33933: connect: Connection refused (error 111)
I20260928 14:38:34.108515 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 20893
I20260928 14:38:34.109897 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.108656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37691 (local address 127.0.0.1:51694)
0928 14:38:34.109715 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.109786 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.194:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":954}
I20260928 14:38:34.114368 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 20530
I20260928 14:38:34.121229 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:38565
--webserver_interface=127.6.23.254
--webserver_port=37865
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:38565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:34.236194 21059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:34.236490 21059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:34.236531 21059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:34.240572 21059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:34.240643 21059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:34.240664 21059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:34.240680 21059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:34.240696 21059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:34.246186 21059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:38565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:38565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=37865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:34.247331 21059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:34.248386 21059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:34.257109 21065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:34.257112 21064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:34.257354 21059 server_base.cc:1034] running on GCE node
W20260928 14:38:34.257159 21067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:34.258081 21059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:34.259191 21059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:34.260406 21059 hybrid_clock.cc:648] HybridClock initialized: now 1790606314260389 us; error 32 us; skew 500 ppm
Sep 28 14:38:34 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606314, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:34.263161 21059 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:38:34.264082 21059 webserver.cc:460] Webserver started at http://127.6.23.254:37865/ using document root <none> and password file <none>
I20260928 14:38:34.264606 21059 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:34.264665 21059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:34.268712 21059 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:38:34.270906 21074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:34.272251 21059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:38:34.272382 21059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "3a10326e298041d397dc104bff976c7a"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "d3678809c8b0d8fb0a57434a1d77ee22"
server_key_iv: "924c87b2efa8adabc93d3f1bd6514d64"
server_key_version: "encryptionkey@0"
I20260928 14:38:34.272934 21059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:34.302695 21059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:34.306202 21059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:34.306432 21059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:34.314005 21059 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:38565
I20260928 14:38:34.314314 21126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:38565 every 8 connection(s)
I20260928 14:38:34.315205 21059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:38:34.318475 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 21059
I20260928 14:38:34.319015 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:33933
--local_ip_for_outbound_sockets=127.6.23.193
--tserver_master_addrs=127.6.23.254:38565
--webserver_port=38933
--webserver_interface=127.6.23.193
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:38:34.320134 21127 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:38:34.321916 21127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Bootstrap starting.
I20260928 14:38:34.339993 21127 log.cc:824] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:34.351611 21127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:38:34.352046 21127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: Bootstrap complete.
I20260928 14:38:34.356280 21127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:34.356746 21127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a10326e298041d397dc104bff976c7a, State: Initialized, Role: FOLLOWER
I20260928 14:38:34.357266 21127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:34.357419 21127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:38:34.357518 21127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:38:34.357643 21127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:38:34.360240 21127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:34.360651 21127 leader_election.cc:302] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a10326e298041d397dc104bff976c7a; no voters:
I20260928 14:38:34.361208 21127 leader_election.cc:288] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:38:34.361356 21131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:38:34.361608 21131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [term 3 LEADER]: Becoming Leader. State: Replica: 3a10326e298041d397dc104bff976c7a, State: Running, Role: LEADER
I20260928 14:38:34.361941 21131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } }
I20260928 14:38:34.362558 21127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:34.363478 21132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3a10326e298041d397dc104bff976c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } } }
I20260928 14:38:34.363615 21132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:34.363914 21133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a10326e298041d397dc104bff976c7a. Latest consensus state: current_term: 3 leader_uuid: "3a10326e298041d397dc104bff976c7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a10326e298041d397dc104bff976c7a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 38565 } } }
I20260928 14:38:34.364008 21133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:34.368036 21147 catalog_manager.cc:1259] Loaded cluster ID: ba768ef0a3694d9ab92ace02dbb61fd3
I20260928 14:38:34.368111 21147 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: loading cluster ID for follower catalog manager: success
W20260928 14:38:34.369208 21147 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 3a10326e298041d397dc104bff976c7a: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260928 14:38:34.369298 21147 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260928 14:38:34.369390 21140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:38:34.372548 21140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:38:34.373126 21140 catalog_manager.cc:1259] Loaded cluster ID: ba768ef0a3694d9ab92ace02dbb61fd3
I20260928 14:38:34.373198 21140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 14:38:34.373908 21140 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f56828e99ad google::LogMessage::Fail() at ??:0
@ 0x7f56828ec88a google::LogMessage::SendToLog() at ??:0
@ 0x7f56828e94de google::LogMessage::Flush() at ??:0
@ 0x7f56828eaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f568d22cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f568d22dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f568d22a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f568d26e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5684b3b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5684b3bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5684b3d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5684b2c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f56861706db start_thread at ??:0
@ 0x7f56813e761f clone at ??:0
W20260928 14:38:34.465655 21129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:34.465962 21129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:34.466009 21129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:34.470335 21129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:34.470412 21129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:34.470479 21129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:38:34.476652 21129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:33933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=38933
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:34.477862 21129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:34.478951 21129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:34.489182 21168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:34.489374 21129 server_base.cc:1034] running on GCE node
W20260928 14:38:34.489168 21169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:34.489187 21171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:34.490080 21129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:34.490793 21129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:34.491973 21129 hybrid_clock.cc:648] HybridClock initialized: now 1790606314491940 us; error 46 us; skew 500 ppm
Sep 28 14:38:34 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606314, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:34.494886 21129 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:38:34.495885 21129 webserver.cc:460] Webserver started at http://127.6.23.193:38933/ using document root <none> and password file <none>
I20260928 14:38:34.496477 21129 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:34.496582 21129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:34.500983 21129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:38:34.504233 21178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:34.505960 21129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:38:34.506122 21129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c"
format_stamp: "Formatted at 2026-09-28 14:38:31 on dist-test-slave-06lh"
server_key: "18a24d3950cac92c09860790b7a88704"
server_key_iv: "d5069ee94f209acb5c004b93b19ccfbd"
server_key_version: "encryptionkey@0"
I20260928 14:38:34.506755 21129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:34.526350 21129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:34.530464 21129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:34.530691 21129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:34.531714 21129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:34.533185 21185 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:38:34.537607 21129 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:38:34.537700 21129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260928 14:38:34.537774 21129 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:38:34.540321 21129 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:38:34.540391 21129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260928 14:38:34.540799 21185 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Bootstrap starting.
I20260928 14:38:34.553864 21129 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:33933
I20260928 14:38:34.553952 21293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:33933 every 8 connection(s)
I20260928 14:38:34.555166 21129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:38:34.557933 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 21129
I20260928 14:38:34.558565 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:37691
--local_ip_for_outbound_sockets=127.6.23.194
--tserver_master_addrs=127.6.23.254:38565
--webserver_port=40129
--webserver_interface=127.6.23.194
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:38:34.562211 21185 log.cc:824] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:34.570407 21185 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:38:34.570844 21185 tablet_bootstrap.cc:492] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Bootstrap complete.
I20260928 14:38:34.572052 21185 ts_tablet_manager.cc:1397] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.012s
I20260928 14:38:34.574724 21185 raft_consensus.cc:348] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:34.575209 21185 raft_consensus.cc:729] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 585b9c9bfbb1447ea05b2db1f1bfbd2c, State: Initialized, Role: FOLLOWER
I20260928 14:38:34.575783 21185 consensus_queue.cc:260] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:34.577093 21185 ts_tablet_manager.cc:1428] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 14:38:34.640479 21296 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:38:34.557704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:40695)
0928 14:38:34.558305 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.558377 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:34.559575 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:34.639392 (+ 79817us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":158,"threads_started":1}
W20260928 14:38:34.641131 21294 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:38565 (0 consecutive failures): Network error: Failed to ping master at 127.6.23.254:38565: Client connection negotiation failed: client connection to 127.6.23.254:38565: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:38:34.641963 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.641654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:35245)
0928 14:38:34.641805 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.641889 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:38:34.642663 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.642437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:41851)
0928 14:38:34.642552 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.642609 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 14:38:34.642895 21294 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:38:34.685081 21297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:34.685524 21297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:34.685603 21297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:34.692040 21297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:34.692138 21297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:34.692248 21297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:38:34.698400 21297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:37691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=40129
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:34.699676 21297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:34.700778 21297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:38:34.710983 21297 server_base.cc:1034] running on GCE node
W20260928 14:38:34.710937 21305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:34.710898 21304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:34.710898 21307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:34.711620 21297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:34.712308 21297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:34.713506 21297 hybrid_clock.cc:648] HybridClock initialized: now 1790606314713469 us; error 54 us; skew 500 ppm
Sep 28 14:38:34 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606314, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:34.716409 21297 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:38:34.717352 21297 webserver.cc:460] Webserver started at http://127.6.23.194:40129/ using document root <none> and password file <none>
I20260928 14:38:34.717950 21297 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:34.718057 21297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:34.722342 21297 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:38:34.724617 21314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:34.725955 21297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:38:34.726114 21297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "dabcee39ccd945ef92f4c9edd826d45c"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "87b6e196c9486f8d682742219f7c45a8"
server_key_iv: "9030694fdc42236cc4307bf8f04cd202"
server_key_version: "encryptionkey@0"
I20260928 14:38:34.726732 21297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:34.745244 21297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:34.748366 21297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:34.748595 21297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:34.749254 21297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:34.750130 21297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:34.750201 21297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:34.750277 21297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:34.750320 21297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:34.759740 21297 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:37691
I20260928 14:38:34.759761 21427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:37691 every 8 connection(s)
I20260928 14:38:34.760818 21297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:38:34.761369 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 21297
I20260928 14:38:34.761848 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:33375
--local_ip_for_outbound_sockets=127.6.23.195
--tserver_master_addrs=127.6.23.254:38565
--webserver_port=38567
--webserver_interface=127.6.23.195
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:38391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:38:34.765082 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.763098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:34625)
0928 14:38:34.763993 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.764152 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":175,"threads_started":1}
W20260928 14:38:34.765704 21428 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:38565 (0 consecutive failures): Network error: Failed to ping master at 127.6.23.254:38565: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
I20260928 14:38:34.766427 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.766192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:33393)
0928 14:38:34.766311 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.766373 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:38:34.767153 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.766868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:38177)
0928 14:38:34.767002 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.767068 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:38:34.767396 21428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:38:34.876916 21430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:34.877241 21430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:34.877288 21430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:34.881502 21430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:34.881582 21430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:34.881644 21430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:38:34.887192 21430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:33375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=38567
--tserver_master_addrs=127.6.23.254:38565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:34.888432 21430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:34.889537 21430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:34.899927 21439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:34.900023 21430 server_base.cc:1034] running on GCE node
W20260928 14:38:34.899912 21437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:34.899919 21436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:34.900602 21430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:34.901281 21430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:34.902443 21430 hybrid_clock.cc:648] HybridClock initialized: now 1790606314902407 us; error 50 us; skew 500 ppm
Sep 28 14:38:34 dist-test-slave-06lh krb5kdc[19983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606314, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:34.905388 21430 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:38:34.906351 21430 webserver.cc:460] Webserver started at http://127.6.23.195:38567/ using document root <none> and password file <none>
I20260928 14:38:34.906946 21430 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:34.907053 21430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:34.911162 21430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 14:38:34.913522 21446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:34.914769 21430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:38:34.914913 21430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "310344369ce749cda4cad0f178cca760"
format_stamp: "Formatted at 2026-09-28 14:38:32 on dist-test-slave-06lh"
server_key: "fac4eca29fb96808f75c67b0c2987e61"
server_key_iv: "a437456ad70459b7799a1e67f6b3e009"
server_key_version: "encryptionkey@0"
I20260928 14:38:34.915565 21430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:34.938130 21430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:34.941900 21430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:34.942143 21430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:34.942811 21430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:34.943755 21430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:34.943843 21430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:34.943921 21430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:34.943982 21430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:34.954113 21430 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:33375
I20260928 14:38:34.954136 21559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:33375 every 8 connection(s)
I20260928 14:38:34.955295 21430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:38:34.959396 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.957665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:34613)
0928 14:38:34.958431 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.958554 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":117,"threads_started":1}
I20260928 14:38:34.959609 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 21430
W20260928 14:38:34.959959 21560 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:38565 (0 consecutive failures): Network error: Failed to ping master at 127.6.23.254:38565: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
I20260928 14:38:34.960731 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.960426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:42347)
0928 14:38:34.960591 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.960655 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 14:38:34.961407 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:34.961199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:41051)
0928 14:38:34.961308 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:34.961359 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:38:34.961570 21560 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:38:35.644119 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:35.643645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:51745)
0928 14:38:35.643946 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:35.644012 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
I20260928 14:38:35.768509 21565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:35.768024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:36785)
0928 14:38:35.768306 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:35.768396 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20260928 14:38:35.962699 21566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:35.962226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:55127)
0928 14:38:35.962521 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:35.962589 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":76,"threads_started":1}
I20260928 14:38:35.978386 21567 raft_consensus.cc:482] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:35.978649 21567 raft_consensus.cc:504] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:35.979846 21567 leader_election.cc:288] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691), 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
Sep 28 14:38:35 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606314, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.194@KRBTEST.COM
Sep 28 14:38:35 dist-test-slave-06lh krb5kdc[19983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606314, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.195@KRBTEST.COM
I20260928 14:38:35.997433 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:35.980361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:33375 (local address 127.6.23.193:42517)
0928 14:38:35.980685 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:35.980700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:35.980798 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:35.982764 (+ 1966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:35.982770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:35.983438 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:35.984316 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:35.984338 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:35.986456 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:35.986461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:35.988262 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:35.988277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:35.989938 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:35.989940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:35.990007 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:35.990454 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:35.990466 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:35.992475 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:35.994921 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:35.994925 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:35.994927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:35.995219 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:35.995667 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:35.995670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:35.995672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:35.995769 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:35.996431 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:35.996434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:35.996625 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 14:38:35.996827 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 14:38:35.997232 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":288,"spinlock_wait_cycles":40192,"thread_start_us":89,"threads_started":1}
I20260928 14:38:35.997479 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:35.980272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:43491 (local address 127.6.23.194:37691)
0928 14:38:35.980751 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:35.980779 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:35.980829 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:38:35.981770 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:35.981779 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:35.982277 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:35.982648 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:35.984540 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:35.986196 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:35.988455 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:35.989222 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:35.989432 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:35.992458 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:35.992467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:35.992485 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:35.992526 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:35.994281 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:35.994954 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:35.994960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:35.994967 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:35.995126 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:35.995685 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:35.995689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:35.995690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:35.996188 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:35.996296 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:35.996958 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 14:38:35.997232 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":143,"threads_started":1}
I20260928 14:38:35.997479 21564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:35.980080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37691 (local address 127.6.23.193:43491)
0928 14:38:35.980223 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:35.980237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:35.980366 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:35.982764 (+ 2398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:35.982770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:35.983438 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:35.984316 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:35.984338 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:35.986456 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:35.986461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:35.988262 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:35.988277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:35.989413 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:35.989420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:35.989620 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:35.990342 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:35.990369 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:35.992131 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:35.994410 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:35.994420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:35.994436 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:35.994793 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:35.995311 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:35.995317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:35.995322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:35.995552 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:35.996313 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:35.996320 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:35.996626 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 14:38:35.996827 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 14:38:35.997321 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":537,"spinlock_wait_cycles":5120}
I20260928 14:38:35.997473 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:35.980551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:42517 (local address 127.6.23.195:33375)
0928 14:38:35.980879 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:35.980899 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:35.980941 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:38:35.981770 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:35.981779 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:35.982277 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:35.982648 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:35.984540 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:35.986196 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:35.988455 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:35.989222 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:35.989416 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:35.992663 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:35.992667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:35.992678 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:35.992707 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:35.994771 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:35.995375 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:35.995384 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:35.995397 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:35.995559 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:35.995872 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:35.995875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:35.995876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:35.996322 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:35.996478 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:35.996959 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 14:38:35.997256 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
W20260928 14:38:36.000137 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
W20260928 14:38:36.000398 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
I20260928 14:38:36.000478 21180 leader_election.cc:302] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585b9c9bfbb1447ea05b2db1f1bfbd2c; no voters: 310344369ce749cda4cad0f178cca760, dabcee39ccd945ef92f4c9edd826d45c
I20260928 14:38:36.000790 21567 raft_consensus.cc:2726] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:38:36.645373 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:36.644935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:38873)
0928 14:38:36.645216 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:36.645276 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20260928 14:38:36.769783 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:36.769316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:46533)
0928 14:38:36.769620 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:36.769685 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":78,"threads_started":1}
I20260928 14:38:36.963948 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:36.963551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:60975)
0928 14:38:36.963801 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:36.963853 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260928 14:38:37.646749 21574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:37.646286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:33429)
0928 14:38:37.646582 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:37.646650 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
I20260928 14:38:37.771155 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:37.770749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:40261)
0928 14:38:37.771007 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:37.771060 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260928 14:38:37.901289 21576 raft_consensus.cc:482] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:37.901417 21576 raft_consensus.cc:504] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:37.901865 21576 leader_election.cc:288] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691), 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
W20260928 14:38:37.902796 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
W20260928 14:38:37.902985 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
I20260928 14:38:37.903049 21180 leader_election.cc:302] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585b9c9bfbb1447ea05b2db1f1bfbd2c; no voters: 310344369ce749cda4cad0f178cca760, dabcee39ccd945ef92f4c9edd826d45c
I20260928 14:38:37.903218 21576 raft_consensus.cc:2726] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:38:37.965287 21577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:37.964827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:57419)
0928 14:38:37.965126 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:37.965191 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":106,"threads_started":1}
I20260928 14:38:38.648128 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:38.647681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:41841)
0928 14:38:38.647953 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:38.648028 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":84,"threads_started":1}
I20260928 14:38:38.772471 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:38.772016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:56235)
0928 14:38:38.772305 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:38.772374 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260928 14:38:38.966547 21580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:38.966123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:37969)
0928 14:38:38.966381 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:38.966439 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260928 14:38:39.649748 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:39.649174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:49157)
0928 14:38:39.649576 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:39.649647 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":92,"threads_started":1}
I20260928 14:38:39.773780 21582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:39.773331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:56891)
0928 14:38:39.773619 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:39.773681 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":96,"threads_started":1}
I20260928 14:38:39.935397 21583 raft_consensus.cc:482] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:39.935545 21583 raft_consensus.cc:504] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:39.936004 21583 leader_election.cc:288] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691), 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
W20260928 14:38:39.936947 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
W20260928 14:38:39.937091 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
I20260928 14:38:39.937153 21180 leader_election.cc:302] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585b9c9bfbb1447ea05b2db1f1bfbd2c; no voters: 310344369ce749cda4cad0f178cca760, dabcee39ccd945ef92f4c9edd826d45c
I20260928 14:38:39.937338 21583 raft_consensus.cc:2726] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:38:39.968065 21584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:39.967608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:60273)
0928 14:38:39.967900 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:39.967966 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260928 14:38:40.651126 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:40.650657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:39965)
0928 14:38:40.650961 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:40.651028 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
I20260928 14:38:40.775202 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:40.774726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:43565)
0928 14:38:40.775039 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:40.775102 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260928 14:38:40.969444 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:40.968955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:60515)
0928 14:38:40.969268 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:40.969345 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
I20260928 14:38:41.652609 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:41.652087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:47815)
0928 14:38:41.652426 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:41.652505 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260928 14:38:41.776636 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:41.776123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:40737)
0928 14:38:41.776465 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:41.776538 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":75,"threads_started":1}
I20260928 14:38:41.970880 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:41.970345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:51003)
0928 14:38:41.970707 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:41.970771 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":104,"threads_started":1}
I20260928 14:38:42.653966 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:42.653493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:49421)
0928 14:38:42.653786 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:42.653862 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":93,"threads_started":1}
I20260928 14:38:42.777899 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:42.777484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:50987)
0928 14:38:42.777738 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:42.777800 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260928 14:38:42.972167 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:42.971737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:50687)
0928 14:38:42.972012 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:42.972068 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260928 14:38:43.464864 21594 raft_consensus.cc:482] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:43.465044 21594 raft_consensus.cc:504] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:43.465487 21594 leader_election.cc:288] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691), 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
W20260928 14:38:43.466401 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
W20260928 14:38:43.466567 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
I20260928 14:38:43.466645 21180 leader_election.cc:302] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585b9c9bfbb1447ea05b2db1f1bfbd2c; no voters: 310344369ce749cda4cad0f178cca760, dabcee39ccd945ef92f4c9edd826d45c
I20260928 14:38:43.467159 21594 raft_consensus.cc:2726] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:38:43.655285 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:43.654808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:43021)
0928 14:38:43.655071 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:43.655126 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
I20260928 14:38:43.779114 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:43.778695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:43075)
0928 14:38:43.778954 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:43.779017 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260928 14:38:43.973531 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:43.973079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:33283)
0928 14:38:43.973368 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:43.973432 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260928 14:38:44.656791 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:44.656276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:57743)
0928 14:38:44.656621 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:44.656691 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
I20260928 14:38:44.780512 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:44.780032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:60753)
0928 14:38:44.780358 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:44.780419 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
I20260928 14:38:44.974972 21600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:44.974491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:53195)
0928 14:38:44.974796 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:44.974868 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":84,"threads_started":1}
I20260928 14:38:45.658118 21601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:45.657679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:41371)
0928 14:38:45.657954 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:45.658018 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260928 14:38:45.781900 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:45.781462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:55065)
0928 14:38:45.781741 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:45.781800 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260928 14:38:45.976332 21603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:45.975879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:60461)
0928 14:38:45.976168 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:45.976231 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260928 14:38:46.659427 21604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:46.658904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:51393)
0928 14:38:46.659196 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:46.659321 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260928 14:38:46.783098 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:46.782645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:35079)
0928 14:38:46.782943 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:46.783003 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260928 14:38:46.977541 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:46.977148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:32771)
0928 14:38:46.977390 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:46.977442 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":71,"threads_started":1}
I20260928 14:38:47.660840 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:47.660337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:59781)
0928 14:38:47.660629 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:47.660696 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
I20260928 14:38:47.784394 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:47.783901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:50237)
0928 14:38:47.784171 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:47.784270 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260928 14:38:47.978871 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:47.978393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:39179)
0928 14:38:47.978683 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:47.978758 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260928 14:38:48.241366 21610 raft_consensus.cc:482] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:48.241511 21610 raft_consensus.cc:504] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabcee39ccd945ef92f4c9edd826d45c" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37691 } } peers { permanent_uuid: "310344369ce749cda4cad0f178cca760" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 33375 } } peers { permanent_uuid: "585b9c9bfbb1447ea05b2db1f1bfbd2c" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 33933 } }
I20260928 14:38:48.241989 21610 leader_election.cc:288] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691), 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375)
W20260928 14:38:48.242897 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabcee39ccd945ef92f4c9edd826d45c (127.6.23.194:37691): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
W20260928 14:38:48.243057 21180 leader_election.cc:341] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 310344369ce749cda4cad0f178cca760 (127.6.23.195:33375): Not found: Tablet not found: 03add453fca64c5b910f53d63b0f57db
I20260928 14:38:48.243120 21180 leader_election.cc:302] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 585b9c9bfbb1447ea05b2db1f1bfbd2c; no voters: 310344369ce749cda4cad0f178cca760, dabcee39ccd945ef92f4c9edd826d45c
I20260928 14:38:48.243350 21610 raft_consensus.cc:2726] T 03add453fca64c5b910f53d63b0f57db P 585b9c9bfbb1447ea05b2db1f1bfbd2c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:38:48.662209 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:48.661698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:44109)
0928 14:38:48.662026 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:48.662098 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":114,"threads_started":1}
I20260928 14:38:48.785832 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:48.785327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:40885)
0928 14:38:48.785639 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:48.785724 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
I20260928 14:38:48.980232 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:48.979796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.195:49801)
0928 14:38:48.980072 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:48.980134 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260928 14:38:49.663807 21614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:49.663353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.193:52865)
0928 14:38:49.663650 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:49.663710 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
I20260928 14:38:49.787093 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:49.786716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:38565 (local address 127.6.23.194:37143)
0928 14:38:49.786948 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:49.786999 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:38565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260928 14:38:49.960558 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 21129
I20260928 14:38:49.966380 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 21297
I20260928 14:38:49.971927 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 21430
2026-09-28T14:38:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19609 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:38:50 dist-test-slave-06lh krb5kdc[21618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:38:50 dist-test-slave-06lh krb5kdc[21618](info): set up 2 sockets
Sep 28 14:38:50 dist-test-slave-06lh krb5kdc[21618](info): commencing operation
krb5kdc: starting...
W20260928 14:38:51.019965 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:38:51 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:38:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:38:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:38:51.147217 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45523
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:51.265185 21631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:51.265566 21631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:51.265648 21631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:51.269873 21631 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:38:51.269984 21631 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:38:51.270021 21631 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:38:51.270053 21631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:38:51.270094 21631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:51.270138 21631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:51.270174 21631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:38:51.270206 21631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:38:51.275727 21631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:51.276901 21631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:51.277987 21631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:38:51.287388 21631 server_base.cc:1034] running on GCE node
W20260928 14:38:51.287343 21636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:51.287343 21637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:51.287348 21639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:51.288383 21631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:51.289615 21631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:51.290783 21631 hybrid_clock.cc:648] HybridClock initialized: now 1790606331290757 us; error 41 us; skew 500 ppm
Sep 28 14:38:51 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:51.293794 21631 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:38:51.294796 21631 webserver.cc:460] Webserver started at http://127.6.23.254:44901/ using document root <none> and password file <none>
I20260928 14:38:51.295475 21631 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:51.295547 21631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:51.295785 21631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:51.297957 21631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "78c9cf0225e87744a7adc3976292331b"
server_key_iv: "fb10705edef90937b4572a1ee7f1d779"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.298595 21631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "78c9cf0225e87744a7adc3976292331b"
server_key_iv: "fb10705edef90937b4572a1ee7f1d779"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.302554 21631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:38:51.305233 21646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:51.306569 21631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:38:51.306725 21631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "78c9cf0225e87744a7adc3976292331b"
server_key_iv: "fb10705edef90937b4572a1ee7f1d779"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.306851 21631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:51.329567 21631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:51.333374 21631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:51.333613 21631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:51.341698 21631 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:45523
I20260928 14:38:51.342097 21698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:45523 every 8 connection(s)
I20260928 14:38:51.343057 21631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:38:51.344079 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 21631
I20260928 14:38:51.344214 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:38:51.344530 6239 external_mini_cluster.cc:1428] Setting key 52e3e5280fc25d6e8d87e9bd48b81931
I20260928 14:38:51.347088 21699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:51.353503 21699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Bootstrap starting.
Sep 28 14:38:51 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:51.356091 21699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:51.356884 21699 log.cc:824] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:51.359110 21699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: No bootstrap required, opened a new log
I20260928 14:38:51.359764 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:51.346222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.6.23.254:45523)
0928 14:38:51.346738 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:51.346754 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:51.346793 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:38:51.347628 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:51.347651 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:51.348016 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:51.348401 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:51.349259 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.350271 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.351200 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.351762 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.351940 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:51.354607 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:51.354613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:51.354628 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:51.354659 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:51.356563 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:51.357087 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:51.357098 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:51.357106 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:51.357284 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:51.357591 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:51.357594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:51.357596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:51.358037 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:51.358238 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:51.358566 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 14:38:51.358891 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":173,"threads_started":1}
I20260928 14:38:51.363471 21699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:38:51.363751 21699 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:51.363839 21699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f0624b724c4bb1ac9af0c13d8ff8b1, State: Initialized, Role: FOLLOWER
I20260928 14:38:51.364343 21699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:38:51.364504 21699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:38:51.364591 21699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:38:51.364691 21699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:51.365746 21699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:38:51.366122 21699 leader_election.cc:302] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16f0624b724c4bb1ac9af0c13d8ff8b1; no voters:
I20260928 14:38:51.366432 21699 leader_election.cc:288] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:38:51.366600 21704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:51.366815 21704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 1 LEADER]: Becoming Leader. State: Replica: 16f0624b724c4bb1ac9af0c13d8ff8b1, State: Running, Role: LEADER
I20260928 14:38:51.367188 21704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:38:51.367839 21699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:38:51.368907 21705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } } }
I20260928 14:38:51.369032 21705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:51.369324 21706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16f0624b724c4bb1ac9af0c13d8ff8b1. Latest consensus state: current_term: 1 leader_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } } }
I20260928 14:38:51.369441 21713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:38:51.369498 21706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:38:51.373485 21713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:38:51.380036 21713 catalog_manager.cc:1347] Generated new cluster ID: 4b77008b3a16454bafc28bd49dcdbccc
I20260928 14:38:51.380136 21713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:38:51.404608 21713 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:38:51.405679 21713 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:38:51.416931 21713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Generated new TSK 0
I20260928 14:38:51.417730 21713 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:38:51.455147 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45523
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:51.571506 21725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:51.571816 21725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:51.571895 21725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:51.576004 21725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:51.576099 21725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:51.576195 21725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:38:51.581426 21725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:51.582588 21725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:51.583743 21725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:38:51.593830 21725 server_base.cc:1034] running on GCE node
W20260928 14:38:51.593740 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:51.593736 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:51.593744 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:51.594456 21725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:51.595165 21725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:51.596376 21725 hybrid_clock.cc:648] HybridClock initialized: now 1790606331596348 us; error 44 us; skew 500 ppm
Sep 28 14:38:51 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:51.599185 21725 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:38:51.600162 21725 webserver.cc:460] Webserver started at http://127.6.23.193:34953/ using document root <none> and password file <none>
I20260928 14:38:51.600806 21725 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:51.600903 21725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:51.601145 21725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:51.603348 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "404165dc8c0c4073b1a25402534d3bdb"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "b3eaa67dd960254738a51807e4d4dcef"
server_key_iv: "91a3d2efa1cf6b2729088282907557ac"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.603947 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "404165dc8c0c4073b1a25402534d3bdb"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "b3eaa67dd960254738a51807e4d4dcef"
server_key_iv: "91a3d2efa1cf6b2729088282907557ac"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.607918 21725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:38:51.610656 21740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:51.612056 21725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:38:51.612205 21725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "404165dc8c0c4073b1a25402534d3bdb"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "b3eaa67dd960254738a51807e4d4dcef"
server_key_iv: "91a3d2efa1cf6b2729088282907557ac"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.612339 21725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:51.632726 21725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:51.635882 21725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:51.636108 21725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:51.636750 21725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:51.637625 21725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:51.637698 21725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:51.637773 21725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:51.637815 21725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:51.646859 21725 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:35473
I20260928 14:38:51.646883 21853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:35473 every 8 connection(s)
I20260928 14:38:51.647850 21725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:38:51.652634 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 21725
I20260928 14:38:51.652778 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:38:51.653110 6239 external_mini_cluster.cc:1428] Setting key 99c08c57f34a0f6d128f322dcefef6c5
Sep 28 14:38:51 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:38:51.664081 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:51.649786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:58701 (local address 127.6.23.254:45523)
0928 14:38:51.649966 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:51.649971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:51.651040 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 14:38:51.651957 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:51.651960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:51.652008 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:51.652091 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:51.653707 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.654460 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.656363 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.656778 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.656936 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:51.659801 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:51.659808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:51.659814 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:51.659859 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:51.661600 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:51.662152 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:51.662159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:51.662162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:51.662259 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:51.662687 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:51.662691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:51.662694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:51.662970 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:51.663086 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:51.663706 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 14:38:51.663875 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:38:51.664963 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:51.650218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:58701)
0928 14:38:51.650886 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:51.650929 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:51.651751 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:51.652257 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:51.652264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:51.652834 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:51.653540 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:51.653559 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.654632 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.654635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:51.656193 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:51.656209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.656975 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.656978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:51.657133 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:51.657836 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:51.657854 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:51.659604 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:51.661739 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:51.661745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:51.661759 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:51.662028 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:51.662407 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:51.662410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:51.662412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:51.662567 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:51.663082 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:51.663087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:51.663442 (+ 355us) client_negotiation.cc:769] Sending connection context
0928 14:38:51.663611 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 14:38:51.663987 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":120,"threads_started":1}
I20260928 14:38:51.666481 21854 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:38:51.666782 21854 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:51.667421 21854 heartbeater.cc:511] Master 127.6.23.254:45523 requested a full tablet report, sending...
I20260928 14:38:51.669070 21663 ts_manager.cc:194] Registered new tserver with Master: 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473)
I20260928 14:38:51.670872 21663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:58701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:38:51.682483 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45523
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:51.795842 21859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:51.796123 21859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:51.796206 21859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:51.800371 21859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:51.800459 21859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:51.800559 21859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:38:51.806062 21859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:51.807310 21859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:51.808393 21859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:51.818413 21865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:51.818413 21864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:51.818405 21867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:51.818815 21859 server_base.cc:1034] running on GCE node
I20260928 14:38:51.819309 21859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:51.819975 21859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:51.821139 21859 hybrid_clock.cc:648] HybridClock initialized: now 1790606331821124 us; error 28 us; skew 500 ppm
Sep 28 14:38:51 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:51.823896 21859 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:38:51.824824 21859 webserver.cc:460] Webserver started at http://127.6.23.194:35299/ using document root <none> and password file <none>
I20260928 14:38:51.825359 21859 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:51.825428 21859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:51.825598 21859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:51.827749 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "1e415d5890284636b527334ecfeb9e9e"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "a9f302fe40a09f2b8a5833f3ae27330e"
server_key_iv: "4a8b14ca5b95b68a4bfd2cdff409fdc1"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.828649 21859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "1e415d5890284636b527334ecfeb9e9e"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "a9f302fe40a09f2b8a5833f3ae27330e"
server_key_iv: "4a8b14ca5b95b68a4bfd2cdff409fdc1"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.832680 21859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:38:51.835184 21874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:51.836614 21859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:38:51.836746 21859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "1e415d5890284636b527334ecfeb9e9e"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "a9f302fe40a09f2b8a5833f3ae27330e"
server_key_iv: "4a8b14ca5b95b68a4bfd2cdff409fdc1"
server_key_version: "encryptionkey@0"
I20260928 14:38:51.836867 21859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:51.862849 21859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:51.866045 21859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:51.866267 21859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:51.866938 21859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:51.867949 21859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:51.868000 21859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:51.868069 21859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:51.868109 21859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:51.877667 21859 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:43483
I20260928 14:38:51.877684 21987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:43483 every 8 connection(s)
I20260928 14:38:51.878626 21859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:38:51.879668 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 21859
I20260928 14:38:51.879760 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:38:51.880040 6239 external_mini_cluster.cc:1428] Setting key 83d928d46a8ab501a07219d9840d1924
Sep 28 14:38:51 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606331, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
I20260928 14:38:51.894297 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:51.880557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:54379 (local address 127.6.23.254:45523)
0928 14:38:51.880701 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:51.880704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:51.881978 (+ 1274us) server_negotiation.cc:407] Connection header received
0928 14:38:51.882880 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:51.882884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:51.882938 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:51.883013 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:51.884431 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.884976 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.886878 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.887108 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.887277 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:51.890374 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:51.890377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:51.890379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:51.890405 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:51.891918 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:51.892490 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:51.892493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:51.892494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:51.892568 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:51.892963 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:51.892966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:51.892967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:51.893199 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:51.893277 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:51.893894 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 14:38:51.894043 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:38:51.895205 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:51.881074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:54379)
0928 14:38:51.881828 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:51.881871 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:51.882670 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:51.883185 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:51.883192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:51.883654 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:51.884263 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:51.884286 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.885116 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.885122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:51.886736 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:51.886752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:51.887350 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:51.887357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:51.887609 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:51.888268 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:51.888292 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:51.890176 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:51.892029 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:51.892034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:51.892048 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:51.892363 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:51.892650 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:51.892652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:51.892653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:51.892854 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:51.893303 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:51.893312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:51.893603 (+ 291us) client_negotiation.cc:769] Sending connection context
0928 14:38:51.893813 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 14:38:51.894178 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":171,"threads_started":1}
I20260928 14:38:51.896550 21988 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45523
I20260928 14:38:51.896811 21988 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:51.897320 21988 heartbeater.cc:511] Master 127.6.23.254:45523 requested a full tablet report, sending...
I20260928 14:38:51.898512 21663 ts_manager.cc:194] Registered new tserver with Master: 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483)
I20260928 14:38:51.899134 21663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:54379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:38:51.910880 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45523
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:38:52.027556 21993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:38:52.027863 21993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:38:52.027922 21993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:38:52.032047 21993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:38:52.032115 21993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:38:52.032177 21993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:38:52.037609 21993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:38:52.038671 21993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:38:52.039853 21993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:38:52.050271 21999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:52.050315 21993 server_base.cc:1034] running on GCE node
W20260928 14:38:52.050328 21998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:38:52.050263 22001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:38:52.050922 21993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:38:52.051646 21993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:38:52.052806 21993 hybrid_clock.cc:648] HybridClock initialized: now 1790606332052785 us; error 35 us; skew 500 ppm
Sep 28 14:38:52 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606332, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:38:52.055747 21993 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:38:52.056618 21993 webserver.cc:460] Webserver started at http://127.6.23.195:32931/ using document root <none> and password file <none>
I20260928 14:38:52.057185 21993 fs_manager.cc:359] Metadata directory not provided
I20260928 14:38:52.057245 21993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:38:52.057420 21993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:38:52.059700 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "a3fb9849680e4dd7a83cbab28cf63f9c"
format_stamp: "Formatted at 2026-09-28 14:38:52 on dist-test-slave-06lh"
server_key: "282092ba3d417bf5b7d087a4af1272d5"
server_key_iv: "b85023a196d8156b04881a852cd6295c"
server_key_version: "encryptionkey@0"
I20260928 14:38:52.060231 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "a3fb9849680e4dd7a83cbab28cf63f9c"
format_stamp: "Formatted at 2026-09-28 14:38:52 on dist-test-slave-06lh"
server_key: "282092ba3d417bf5b7d087a4af1272d5"
server_key_iv: "b85023a196d8156b04881a852cd6295c"
server_key_version: "encryptionkey@0"
I20260928 14:38:52.064126 21993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:38:52.066555 22008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:52.067951 21993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:38:52.068094 21993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "a3fb9849680e4dd7a83cbab28cf63f9c"
format_stamp: "Formatted at 2026-09-28 14:38:52 on dist-test-slave-06lh"
server_key: "282092ba3d417bf5b7d087a4af1272d5"
server_key_iv: "b85023a196d8156b04881a852cd6295c"
server_key_version: "encryptionkey@0"
I20260928 14:38:52.068254 21993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:38:52.086862 21993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:38:52.090473 21993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:38:52.090700 21993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:38:52.091391 21993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:38:52.092272 21993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:38:52.092348 21993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:52.092422 21993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:38:52.092474 21993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:38:52.101552 21993 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:45419
I20260928 14:38:52.101574 22121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:45419 every 8 connection(s)
I20260928 14:38:52.102473 21993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:38:52.107766 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 21993
I20260928 14:38:52.107921 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:38:52.108214 6239 external_mini_cluster.cc:1428] Setting key 020ab890176b51df9dfaad8e853858ff
Sep 28 14:38:52 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606332, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:38:52.116008 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.104308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:59955 (local address 127.6.23.254:45523)
0928 14:38:52.104451 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:52.104454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:52.105504 (+ 1050us) server_negotiation.cc:407] Connection header received
0928 14:38:52.106446 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:52.106450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:52.106506 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:52.106622 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:52.107923 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.108559 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.109713 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.109935 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.110052 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.112561 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:52.112565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:52.112567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:52.112591 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:52.113861 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:52.114376 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:52.114380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:52.114382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:52.114468 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:52.114828 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:52.114832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:52.114834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:52.115024 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:52.115137 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:52.115716 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 14:38:52.115844 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:38:52.116873 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.104698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:59955)
0928 14:38:52.105354 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:52.105396 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:52.106235 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:52.106806 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:52.106813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:52.107182 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:38:52.107776 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.107791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.108687 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.108695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.109579 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.109589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.110097 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.110100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.110288 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.110807 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:38:52.110823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:38:52.112379 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:38:52.113985 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:52.113991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:52.114007 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:52.114266 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:52.114583 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:38:52.114586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:38:52.114588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:38:52.114728 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:38:52.115134 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:38:52.115141 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:38:52.115468 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 14:38:52.115664 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 14:38:52.115998 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":108,"threads_started":1}
I20260928 14:38:52.117980 22122 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45523
I20260928 14:38:52.118228 22122 heartbeater.cc:464] Registering TS with master...
I20260928 14:38:52.118718 22122 heartbeater.cc:511] Master 127.6.23.254:45523 requested a full tablet report, sending...
I20260928 14:38:52.119738 21663 ts_manager.cc:194] Registered new tserver with Master: a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419)
I20260928 14:38:52.120316 21663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:59955
I20260928 14:38:52.122279 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:38:52.131831 21702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.124297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.6.23.254:45523)
0928 14:38:52.124440 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:52.124443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:52.124664 (+ 221us) server_negotiation.cc:407] Connection header received
0928 14:38:52.124829 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:52.124831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:52.124880 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:52.124964 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:38:52.125672 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.126191 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.127043 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.127351 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.127462 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.128778 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:38:52.128782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:38:52.128784 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:38:52.128811 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:38:52.130074 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:52.130506 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:52.130510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:52.130512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:52.130600 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:38:52.130881 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:38:52.130885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:38:52.130887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:38:52.131127 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:38:52.131308 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:52.131551 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 14:38:52.131673 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:38:52.134884 21663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:38:52.137887 21663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:38:52.153231 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.147821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.6.23.195:45419)
0928 14:38:52.148393 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:52.148407 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:52.148433 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:38:52.148515 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:52.148522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:52.148739 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:52.149205 (+ 466us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:52.149369 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.150247 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.151627 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.152489 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.152601 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.152755 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:52.152969 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 14:38:52.153080 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":148,"threads_started":1}
I20260928 14:38:52.153223 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.147301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:45419 (local address 127.0.0.1:47510)
0928 14:38:52.148039 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:52.148057 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:52.148150 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:52.148898 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:52.148900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:52.148916 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:52.149023 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.149031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.150396 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.150400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.151487 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.151497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.152626 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.152630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.152743 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.152789 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:38:52.152920 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 14:38:52.153061 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":31,"thread_start_us":100,"threads_started":1}
I20260928 14:38:52.153587 22135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.147724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:35473 (local address 127.0.0.1:44220)
0928 14:38:52.148314 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:52.148327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:52.148410 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:52.149241 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:52.149244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:52.149263 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:52.149360 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.149365 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.150560 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.150563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.152001 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.152010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.152953 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.152956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.153047 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.153068 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:38:52.153117 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 14:38:52.153173 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":249,"thread_start_us":101,"threads_started":1}
I20260928 14:38:52.153880 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.146878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:43483 (local address 127.0.0.1:54876)
0928 14:38:52.147352 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:52.147411 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:52.147557 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:52.148354 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:52.148361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:52.148388 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:52.148558 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.148566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.150004 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.150008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.152001 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.152010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.153279 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.153285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.153413 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.153442 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:38:52.153510 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:38:52.153574 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":84,"thread_start_us":103,"threads_started":1}
I20260928 14:38:52.154009 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.147053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.6.23.194:43483)
0928 14:38:52.147591 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:52.147604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:52.147630 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:38:52.147720 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:52.147726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:52.147960 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:52.148130 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:52.148698 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.149855 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.152135 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.152963 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.153076 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.153209 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:52.153726 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 14:38:52.153858 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":134,"threads_started":1}
I20260928 14:38:52.153913 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.147970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44220 (local address 127.6.23.193:35473)
0928 14:38:52.148660 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:52.148676 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:52.148697 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:38:52.148770 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:52.148775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:52.149091 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:52.149254 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:52.149493 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.150414 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.152150 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.152813 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.153304 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.153462 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:52.153515 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:38:52.153624 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":103,"threads_started":1}
I20260928 14:38:52.156785 21788 tablet_service.cc:1467] Processing CreateTablet for tablet ec075eaf26754b928d0cdf6492df0c32 (DEFAULT_TABLE table=test-table [id=3d8c1b78afdf4d16bc85444ebee70c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:52.156807 22056 tablet_service.cc:1467] Processing CreateTablet for tablet ec075eaf26754b928d0cdf6492df0c32 (DEFAULT_TABLE table=test-table [id=3d8c1b78afdf4d16bc85444ebee70c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:52.156809 21922 tablet_service.cc:1467] Processing CreateTablet for tablet ec075eaf26754b928d0cdf6492df0c32 (DEFAULT_TABLE table=test-table [id=3d8c1b78afdf4d16bc85444ebee70c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:38:52.157797 21788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec075eaf26754b928d0cdf6492df0c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:52.157827 21922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec075eaf26754b928d0cdf6492df0c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:52.157832 22056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec075eaf26754b928d0cdf6492df0c32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:38:52.164558 22139 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Bootstrap starting.
I20260928 14:38:52.165886 22140 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: Bootstrap starting.
I20260928 14:38:52.167376 22139 tablet_bootstrap.cc:654] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:52.167588 22141 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: Bootstrap starting.
I20260928 14:38:52.168376 22139 log.cc:824] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:52.168692 22140 tablet_bootstrap.cc:654] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:52.169572 22140 log.cc:824] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:52.170058 22141 tablet_bootstrap.cc:654] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: Neither blocks nor log segments found. Creating new log.
I20260928 14:38:52.170964 22139 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: No bootstrap required, opened a new log
I20260928 14:38:52.170928 22141 log.cc:824] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:38:52.171196 22139 ts_tablet_manager.cc:1397] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260928 14:38:52.171661 22140 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: No bootstrap required, opened a new log
I20260928 14:38:52.171859 22140 ts_tablet_manager.cc:1397] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:38:52.173101 22141 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: No bootstrap required, opened a new log
I20260928 14:38:52.173269 22141 ts_tablet_manager.cc:1397] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 14:38:52.174955 22139 raft_consensus.cc:348] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.175276 22139 raft_consensus.cc:374] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:52.175349 22139 raft_consensus.cc:729] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404165dc8c0c4073b1a25402534d3bdb, State: Initialized, Role: FOLLOWER
I20260928 14:38:52.175896 22139 consensus_queue.cc:260] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.175794 22140 raft_consensus.cc:348] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.176059 22140 raft_consensus.cc:374] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:52.176107 22140 raft_consensus.cc:729] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e415d5890284636b527334ecfeb9e9e, State: Initialized, Role: FOLLOWER
I20260928 14:38:52.176908 21854 heartbeater.cc:503] Master 127.6.23.254:45523 was elected leader, sending a full tablet report...
I20260928 14:38:52.176987 22141 raft_consensus.cc:348] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.177237 22141 raft_consensus.cc:374] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:38:52.177284 22141 raft_consensus.cc:729] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fb9849680e4dd7a83cbab28cf63f9c, State: Initialized, Role: FOLLOWER
I20260928 14:38:52.177345 22139 ts_tablet_manager.cc:1428] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:38:52.177747 22141 consensus_queue.cc:260] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.177793 22140 consensus_queue.cc:260] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.178416 22122 heartbeater.cc:503] Master 127.6.23.254:45523 was elected leader, sending a full tablet report...
I20260928 14:38:52.178615 21988 heartbeater.cc:503] Master 127.6.23.254:45523 was elected leader, sending a full tablet report...
I20260928 14:38:52.178774 22141 ts_tablet_manager.cc:1428] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260928 14:38:52.179730 22140 ts_tablet_manager.cc:1428] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260928 14:38:52.199550 22146 raft_consensus.cc:482] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:38:52.199772 22146 raft_consensus.cc:504] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.200903 22146 leader_election.cc:288] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483), 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473)
I20260928 14:38:52.204537 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.201199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:46221 (local address 127.6.23.194:43483)
0928 14:38:52.201332 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:52.201336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:52.201350 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:38:52.201475 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:52.201479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:52.201534 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:52.201643 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:52.201929 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.202444 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.203547 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.203968 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.204113 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.204169 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:52.204350 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 14:38:52.204416 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:38:52.204546 22124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.201130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:43483 (local address 127.6.23.195:46221)
0928 14:38:52.201271 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:52.201287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:52.201377 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:52.201659 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:52.201662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:52.201682 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:52.201808 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.201814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.202568 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.202571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.203424 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.203434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.204099 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.204103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.204210 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.204246 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:38:52.204335 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:38:52.204405 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:38:52.205056 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.201426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:35473 (local address 127.6.23.195:36581)
0928 14:38:52.201854 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:52.201864 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 14:38:52.201988 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:38:52.202252 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:38:52.202254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:38:52.202270 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:38:52.202397 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.202402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.203100 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.203107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.203945 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:38:52.203954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.204622 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.204626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:38:52.204723 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.204743 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:38:52.204841 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 14:38:52.204911 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":132,"threads_started":1}
I20260928 14:38:52.205081 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.201609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:36581 (local address 127.6.23.193:35473)
0928 14:38:52.201742 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:52.201747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:52.201953 (+ 206us) server_negotiation.cc:407] Connection header received
0928 14:38:52.202110 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:52.202112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:52.202156 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:52.202234 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:38:52.202512 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.202995 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.204083 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.204489 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.204625 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.204671 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:52.204872 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 14:38:52.204955 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:38:52.205287 21942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec075eaf26754b928d0cdf6492df0c32" candidate_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e415d5890284636b527334ecfeb9e9e" is_pre_election: true
I20260928 14:38:52.205624 21942 raft_consensus.cc:2454] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fb9849680e4dd7a83cbab28cf63f9c in term 0.
I20260928 14:38:52.205755 21808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec075eaf26754b928d0cdf6492df0c32" candidate_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404165dc8c0c4073b1a25402534d3bdb" is_pre_election: true
I20260928 14:38:52.206035 21808 raft_consensus.cc:2454] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fb9849680e4dd7a83cbab28cf63f9c in term 0.
I20260928 14:38:52.206108 22010 leader_election.cc:302] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e415d5890284636b527334ecfeb9e9e, a3fb9849680e4dd7a83cbab28cf63f9c; no voters:
I20260928 14:38:52.206419 22146 raft_consensus.cc:2781] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:38:52.206525 22146 raft_consensus.cc:482] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:38:52.206590 22146 raft_consensus.cc:3037] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:52.207710 22146 raft_consensus.cc:504] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.208096 22146 leader_election.cc:288] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [CANDIDATE]: Term 1 election: Requested vote from peers 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483), 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473)
I20260928 14:38:52.208441 21942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec075eaf26754b928d0cdf6492df0c32" candidate_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e415d5890284636b527334ecfeb9e9e"
I20260928 14:38:52.208498 21808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec075eaf26754b928d0cdf6492df0c32" candidate_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404165dc8c0c4073b1a25402534d3bdb"
I20260928 14:38:52.208575 21942 raft_consensus.cc:3037] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:52.208617 21808 raft_consensus.cc:3037] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:38:52.209903 21808 raft_consensus.cc:2454] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fb9849680e4dd7a83cbab28cf63f9c in term 1.
I20260928 14:38:52.209919 21942 raft_consensus.cc:2454] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fb9849680e4dd7a83cbab28cf63f9c in term 1.
I20260928 14:38:52.210246 22010 leader_election.cc:302] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 404165dc8c0c4073b1a25402534d3bdb, a3fb9849680e4dd7a83cbab28cf63f9c; no voters:
I20260928 14:38:52.210409 22146 raft_consensus.cc:2781] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:38:52.210646 22146 raft_consensus.cc:686] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 LEADER]: Becoming Leader. State: Replica: a3fb9849680e4dd7a83cbab28cf63f9c, State: Running, Role: LEADER
I20260928 14:38:52.210990 22146 consensus_queue.cc:237] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:38:52.214507 21663 catalog_manager.cc:5495] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195). New cstate: current_term: 1 leader_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } health_report { overall_health: HEALTHY } } }
I20260928 14:38:52.232941 22137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.229284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.6.23.195:45419)
0928 14:38:52.229411 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:38:52.229417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:38:52.229432 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:38:52.229552 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:38:52.229555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:38:52.229613 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:38:52.229729 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:38:52.229988 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.230495 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.231432 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:38:52.231660 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:38:52.231769 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:38:52.232106 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:38:52.232544 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:38:52.232686 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:38:52.232727 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 14:38:52.232788 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:38:52.238953 21808 raft_consensus.cc:1261] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Refusing update from remote peer a3fb9849680e4dd7a83cbab28cf63f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:52.238953 21942 raft_consensus.cc:1261] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 1 FOLLOWER]: Refusing update from remote peer a3fb9849680e4dd7a83cbab28cf63f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:38:52.239849 22149 consensus_queue.cc:1035] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:52.240137 22146 consensus_queue.cc:1035] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:38:52.249246 22153 mvcc.cc:205] Tried to move back new op lower bound from 7334323536843427840 to 7334323536739495936. Current Snapshot: MvccSnapshot[applied={T|T < 7334323536843427840}]
I20260928 14:38:52.249611 22152 mvcc.cc:205] Tried to move back new op lower bound from 7334323536843427840 to 7334323536739495936. Current Snapshot: MvccSnapshot[applied={T|T < 7334323536843427840}]
I20260928 14:38:52.249686 22154 mvcc.cc:205] Tried to move back new op lower bound from 7334323536843427840 to 7334323536739495936. Current Snapshot: MvccSnapshot[applied={T|T < 7334323536843427840}]
I20260928 14:38:52.257010 21663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:38:52.257170 21663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41720:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:38:52.259748 21663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Sending DeleteTablet for 3 replicas of tablet ec075eaf26754b928d0cdf6492df0c32
I20260928 14:38:52.260664 21922 tablet_service.cc:1514] Processing DeleteTablet for tablet ec075eaf26754b928d0cdf6492df0c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:52 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:54876
I20260928 14:38:52.260660 22056 tablet_service.cc:1514] Processing DeleteTablet for tablet ec075eaf26754b928d0cdf6492df0c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:52 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:47510
I20260928 14:38:52.261201 22161 tablet_replica.cc:330] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: stopping tablet replica
I20260928 14:38:52.261231 21788 tablet_service.cc:1514] Processing DeleteTablet for tablet ec075eaf26754b928d0cdf6492df0c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:52 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:44220
I20260928 14:38:52.261358 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 21725
I20260928 14:38:52.262038 22161 raft_consensus.cc:2229] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:38:52.262549 22161 raft_consensus.cc:2258] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:52.262769 22162 tablet_replica.cc:330] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: stopping tablet replica
I20260928 14:38:52.263108 22162 raft_consensus.cc:2229] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:38:52.263808 22162 raft_consensus.cc:2258] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:38:52.264575 22161 ts_tablet_manager.cc:1905] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:52.265762 22162 ts_tablet_manager.cc:1905] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:38:52.267621 22161 ts_tablet_manager.cc:1918] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:52.267834 22161 log.cc:1196] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/ec075eaf26754b928d0cdf6492df0c32
I20260928 14:38:52.268245 22161 ts_tablet_manager.cc:1939] T ec075eaf26754b928d0cdf6492df0c32 P a3fb9849680e4dd7a83cbab28cf63f9c: Deleting consensus metadata
I20260928 14:38:52.269361 22162 ts_tablet_manager.cc:1918] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:38:52.269466 22162 log.cc:1196] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/ec075eaf26754b928d0cdf6492df0c32
I20260928 14:38:52.269848 22162 ts_tablet_manager.cc:1939] T ec075eaf26754b928d0cdf6492df0c32 P 1e415d5890284636b527334ecfeb9e9e: Deleting consensus metadata
I20260928 14:38:52.269797 21649 catalog_manager.cc:4841] TS a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419): tablet ec075eaf26754b928d0cdf6492df0c32 (table test-table [id=3d8c1b78afdf4d16bc85444ebee70c9f]) successfully deleted
W20260928 14:38:52.270330 21648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:35473 (error 108)
I20260928 14:38:52.271263 21648 catalog_manager.cc:4841] TS 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483): tablet ec075eaf26754b928d0cdf6492df0c32 (table test-table [id=3d8c1b78afdf4d16bc85444ebee70c9f]) successfully deleted
I20260928 14:38:52.271389 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:38:52.271068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:35473 (local address 127.0.0.1:44230)
0928 14:38:52.271204 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:38:52.271300 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 14:38:52.271525 21648 catalog_manager.cc:4584] TS 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec075eaf26754b928d0cdf6492df0c32: Network error: Client connection negotiation failed: client connection to 127.6.23.193:35473: connect: Connection refused (error 111)
I20260928 14:38:52.272007 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 21859
I20260928 14:38:52.277699 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 21993
I20260928 14:38:52.282982 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 21631
I20260928 14:39:12.289875 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45523
--webserver_interface=127.6.23.254
--webserver_port=44901
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:12.404727 22164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:12.405077 22164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:12.405162 22164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:12.409355 22164 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:39:12.409439 22164 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:39:12.409468 22164 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:39:12.409528 22164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:39:12.409571 22164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:12.409612 22164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:12.409646 22164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:39:12.409677 22164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:39:12.415213 22164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:12.416446 22164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:12.417560 22164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:12.431581 22170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:12.431620 22171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:12.431669 22173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:12.431869 22164 server_base.cc:1034] running on GCE node
I20260928 14:39:12.432570 22164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:12.434084 22164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:12.435281 22164 hybrid_clock.cc:648] HybridClock initialized: now 1790606352435270 us; error 48 us; skew 500 ppm
Sep 28 14:39:12 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606352, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:12.438356 22164 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:39:12.439453 22164 webserver.cc:460] Webserver started at http://127.6.23.254:44901/ using document root <none> and password file <none>
I20260928 14:39:12.440069 22164 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:12.440186 22164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:12.444294 22164 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:39:12.446600 22180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:12.448000 22164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:39:12.448151 22164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "78c9cf0225e87744a7adc3976292331b"
server_key_iv: "fb10705edef90937b4572a1ee7f1d779"
server_key_version: "encryptionkey@0"
I20260928 14:39:12.448787 22164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:12.472108 22164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:12.475658 22164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:12.475878 22164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:12.483217 22164 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:45523
I20260928 14:39:12.483565 22232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:45523 every 8 connection(s)
I20260928 14:39:12.484421 22164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:39:12.486989 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 22164
I20260928 14:39:12.487500 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:35473
--local_ip_for_outbound_sockets=127.6.23.193
--tserver_master_addrs=127.6.23.254:45523
--webserver_port=34953
--webserver_interface=127.6.23.193
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:39:12.489009 22233 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:39:12.490724 22233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Bootstrap starting.
I20260928 14:39:12.508075 22233 log.cc:824] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:12.516960 22233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:39:12.517351 22233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Bootstrap complete.
I20260928 14:39:12.521515 22233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:39:12.521972 22233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f0624b724c4bb1ac9af0c13d8ff8b1, State: Initialized, Role: FOLLOWER
I20260928 14:39:12.522468 22233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:39:12.522616 22233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:39:12.522712 22233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:39:12.522871 22233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:39:12.524994 22233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:39:12.525345 22233 leader_election.cc:302] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16f0624b724c4bb1ac9af0c13d8ff8b1; no voters:
I20260928 14:39:12.525658 22233 leader_election.cc:288] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:39:12.525815 22237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:39:12.526072 22237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 2 LEADER]: Becoming Leader. State: Replica: 16f0624b724c4bb1ac9af0c13d8ff8b1, State: Running, Role: LEADER
I20260928 14:39:12.526384 22237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:39:12.526968 22233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:39:12.527999 22239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16f0624b724c4bb1ac9af0c13d8ff8b1. Latest consensus state: current_term: 2 leader_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } } }
I20260928 14:39:12.528146 22239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:12.528399 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } } }
I20260928 14:39:12.528477 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:12.528481 22246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:39:12.533638 22246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:39:12.534277 22246 catalog_manager.cc:1259] Loaded cluster ID: 4b77008b3a16454bafc28bd49dcdbccc
I20260928 14:39:12.534339 22246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:39:12.536782 22246 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:39:12.537477 22246 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Loaded TSK: 0
I20260928 14:39:12.549943 22246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Generated new TSK 1
I20260928 14:39:12.550225 22246 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:39:12.607724 22235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:12.607998 22235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:12.608044 22235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:12.612283 22235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:12.612349 22235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:12.612434 22235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:39:12.617734 22235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:35473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=34953
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:12.618793 22235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:12.619905 22235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:12.631160 22260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:12.631179 22261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:12.631526 22235 server_base.cc:1034] running on GCE node
W20260928 14:39:12.631693 22263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:12.632117 22235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:12.632784 22235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:12.633973 22235 hybrid_clock.cc:648] HybridClock initialized: now 1790606352633947 us; error 47 us; skew 500 ppm
Sep 28 14:39:12 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606352, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:12.636978 22235 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:39:12.637935 22235 webserver.cc:460] Webserver started at http://127.6.23.193:34953/ using document root <none> and password file <none>
I20260928 14:39:12.638530 22235 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:12.638644 22235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:12.642771 22235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:39:12.645166 22270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:12.646518 22235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:39:12.646660 22235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "404165dc8c0c4073b1a25402534d3bdb"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "b3eaa67dd960254738a51807e4d4dcef"
server_key_iv: "91a3d2efa1cf6b2729088282907557ac"
server_key_version: "encryptionkey@0"
I20260928 14:39:12.647221 22235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:12.661073 22235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:12.664526 22235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:12.664743 22235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:12.665354 22235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:12.666474 22277 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:39:12.671627 22235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:39:12.671707 22235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 14:39:12.671829 22235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:39:12.673529 22235 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:39:12.673602 22235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 14:39:12.674031 22277 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Bootstrap starting.
I20260928 14:39:12.685268 22235 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:35473
I20260928 14:39:12.685968 22384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:35473 every 8 connection(s)
I20260928 14:39:12.686635 22235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:39:12.694404 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 22235
I20260928 14:39:12.694897 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:43483
--local_ip_for_outbound_sockets=127.6.23.194
--tserver_master_addrs=127.6.23.254:45523
--webserver_port=35299
--webserver_interface=127.6.23.194
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:39:12.696107 22277 log.cc:824] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:39:12 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606352, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:12.704317 22277 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:39:12.704787 22277 tablet_bootstrap.cc:492] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Bootstrap complete.
I20260928 14:39:12.706157 22277 ts_tablet_manager.cc:1397] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.011s
I20260928 14:39:12.706518 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:12.689069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:60111)
0928 14:39:12.690331 (+ 1262us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:12.690413 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:12.691485 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:12.692660 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:12.692668 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:12.693110 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:12.693581 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:12.693596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.695219 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.695222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:12.696158 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:12.696168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.697327 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.697331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:12.697518 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:12.698138 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:12.698157 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:12.699916 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:12.702784 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:12.702790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:12.702803 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:12.703146 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:12.703665 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:12.703668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:12.703669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:12.703833 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:12.704631 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:12.704638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:12.704928 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 14:39:12.705127 (+ 199us) client_negotiation.cc:240] Negotiation successful
0928 14:39:12.705477 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":132,"threads_started":1}
I20260928 14:39:12.707170 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:12.689061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:60111 (local address 127.6.23.254:45523)
0928 14:39:12.689968 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:12.689989 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:12.690554 (+ 565us) server_negotiation.cc:407] Connection header received
0928 14:39:12.691955 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:12.691966 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:12.692276 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:12.692612 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:12.693761 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.694896 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.696355 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.697161 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.697345 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:12.700154 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:12.700163 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:12.700183 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:12.700222 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:12.702644 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:12.703337 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:12.703346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:12.703354 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:12.703540 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:12.703953 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:12.703958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:12.703963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:12.704497 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:12.704648 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:12.705312 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 14:39:12.705668 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":116,"threads_started":1}
I20260928 14:39:12.708678 22277 raft_consensus.cc:348] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:39:12.709182 22277 raft_consensus.cc:729] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404165dc8c0c4073b1a25402534d3bdb, State: Initialized, Role: FOLLOWER
I20260928 14:39:12.709684 22277 consensus_queue.cc:260] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } } peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } }
I20260928 14:39:12.709946 22385 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45523
I20260928 14:39:12.710184 22385 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:12.710693 22385 heartbeater.cc:511] Master 127.6.23.254:45523 requested a full tablet report, sending...
I20260928 14:39:12.710726 22277 ts_tablet_manager.cc:1428] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260928 14:39:12.713328 22197 ts_manager.cc:194] Registered new tserver with Master: 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473)
I20260928 14:39:12.717418 22197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:60111
I20260928 14:39:12.720044 22385 heartbeater.cc:503] Master 127.6.23.254:45523 was elected leader, sending a full tablet report...
Sep 28 14:39:12 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606352, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.193@KRBTEST.COM
I20260928 14:39:12.728356 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:12.716813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.6.23.193:35473)
0928 14:39:12.717221 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:12.717236 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:12.717795 (+ 559us) server_negotiation.cc:407] Connection header received
0928 14:39:12.717859 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:12.717863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:12.718024 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:12.718170 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:12.718854 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.720064 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.721358 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.721860 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.721946 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:12.724538 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:12.724547 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:12.724558 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:12.724581 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:12.726201 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:12.726748 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:12.726753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:12.726754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:12.726832 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:12.727114 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:12.727116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:12.727117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:12.727569 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:12.727701 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:12.728082 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 14:39:12.728226 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":100,"threads_started":1}
I20260928 14:39:12.728410 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:12.716758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:35473 (local address 127.0.0.1:57746)
0928 14:39:12.717477 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:12.717514 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:12.717682 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:12.718169 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:12.718175 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:12.718588 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:12.718722 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:12.718730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.720203 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.720208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:12.721234 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:12.721245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.721978 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.721982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:12.722095 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:12.722665 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:12.722680 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:12.724384 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:12.726317 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:12.726326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:12.726332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:12.726633 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:12.726924 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:12.726927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:12.726929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:12.727025 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:12.727688 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:12.727694 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:12.727824 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 14:39:12.727975 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 14:39:12.728178 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":125,"threads_started":1}
I20260928 14:39:12.729470 22319 tablet_service.cc:1514] Processing DeleteTablet for tablet ec075eaf26754b928d0cdf6492df0c32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:38:52 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:57746
I20260928 14:39:12.730329 22394 tablet_replica.cc:330] stopping tablet replica
I20260928 14:39:12.730634 22394 raft_consensus.cc:2229] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:12.730887 22394 raft_consensus.cc:2258] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:12.731884 22394 ts_tablet_manager.cc:1905] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:12.735558 22394 ts_tablet_manager.cc:1918] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:12.735641 22394 log.cc:1196] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal/wals/ec075eaf26754b928d0cdf6492df0c32
I20260928 14:39:12.735927 22394 ts_tablet_manager.cc:1939] T ec075eaf26754b928d0cdf6492df0c32 P 404165dc8c0c4073b1a25402534d3bdb: Deleting consensus metadata
I20260928 14:39:12.737668 22182 catalog_manager.cc:4841] TS 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473): tablet ec075eaf26754b928d0cdf6492df0c32 (table test-table [id=3d8c1b78afdf4d16bc85444ebee70c9f]) successfully deleted
W20260928 14:39:12.817961 22389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:12.818238 22389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:12.818282 22389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:12.822230 22389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:12.822292 22389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:12.822350 22389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:39:12.827987 22389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:43483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=35299
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:12.829102 22389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:12.830145 22389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:12.840013 22399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:12.840047 22400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:12.840227 22402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:12.840090 22389 server_base.cc:1034] running on GCE node
I20260928 14:39:12.840701 22389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:12.841382 22389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:12.842574 22389 hybrid_clock.cc:648] HybridClock initialized: now 1790606352842543 us; error 48 us; skew 500 ppm
Sep 28 14:39:12 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606352, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:12.845239 22389 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:39:12.846148 22389 webserver.cc:460] Webserver started at http://127.6.23.194:35299/ using document root <none> and password file <none>
I20260928 14:39:12.846720 22389 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:12.846824 22389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:12.850725 22389 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:39:12.852977 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:12.854396 22389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:39:12.854496 22389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "1e415d5890284636b527334ecfeb9e9e"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "a9f302fe40a09f2b8a5833f3ae27330e"
server_key_iv: "4a8b14ca5b95b68a4bfd2cdff409fdc1"
server_key_version: "encryptionkey@0"
I20260928 14:39:12.855021 22389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:12.875883 22389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:12.878953 22389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:12.879117 22389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:12.879768 22389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:12.880573 22389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:12.880617 22389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:12.880657 22389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:12.880672 22389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:12.890069 22389 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:43483
I20260928 14:39:12.890090 22522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:43483 every 8 connection(s)
I20260928 14:39:12.891134 22389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:39:12.894148 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 22389
I20260928 14:39:12.894655 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:45419
--local_ip_for_outbound_sockets=127.6.23.195
--tserver_master_addrs=127.6.23.254:45523
--webserver_port=32931
--webserver_interface=127.6.23.195
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:39:12 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606352, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:12.905153 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:12.892982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:38879 (local address 127.6.23.254:45523)
0928 14:39:12.893126 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:12.893130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:12.894155 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 14:39:12.895263 (+ 1108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:12.895267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:12.895329 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:12.895444 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:12.896637 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.897177 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.898341 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.898591 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.898726 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:12.901527 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:12.901530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:12.901532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:12.901557 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:12.902997 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:12.903602 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:12.903610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:12.903611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:12.903687 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:12.904033 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:12.904038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:12.904040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:12.904218 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:12.904329 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:12.904836 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 14:39:12.904980 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:39:12.906013 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:12.893337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:38879)
0928 14:39:12.893993 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:12.894037 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:12.895023 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:12.895632 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:12.895639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:12.896004 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:12.896484 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:12.896497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.897309 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.897312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:12.898208 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:12.898217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:12.898768 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:12.898772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:12.898956 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:12.899618 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:12.899633 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:12.901335 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:12.903117 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:12.903123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:12.903139 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:12.903485 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:12.903787 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:12.903789 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:12.903791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:12.903936 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:12.904327 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:12.904334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:12.904587 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 14:39:12.904798 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 14:39:12.905115 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":119,"threads_started":1}
I20260928 14:39:12.907368 22523 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45523
I20260928 14:39:12.907614 22523 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:12.908109 22523 heartbeater.cc:511] Master 127.6.23.254:45523 requested a full tablet report, sending...
I20260928 14:39:12.909181 22197 ts_manager.cc:194] Registered new tserver with Master: 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483)
I20260928 14:39:12.909750 22197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:38879
W20260928 14:39:13.011106 22526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:13.011507 22526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:13.011582 22526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:13.015825 22526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:13.015892 22526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:13.015954 22526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:39:13.021353 22526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:45419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=32931
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:13.022406 22526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:13.023693 22526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:13.034335 22531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:13.034530 22526 server_base.cc:1034] running on GCE node
W20260928 14:39:13.034319 22534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:13.034319 22532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:13.035090 22526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:13.035792 22526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:13.036978 22526 hybrid_clock.cc:648] HybridClock initialized: now 1790606353036963 us; error 28 us; skew 500 ppm
Sep 28 14:39:13 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606353, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:13.039844 22526 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:39:13.040755 22526 webserver.cc:460] Webserver started at http://127.6.23.195:32931/ using document root <none> and password file <none>
I20260928 14:39:13.041296 22526 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:13.041365 22526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:13.045177 22526 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:39:13.047325 22541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:13.048628 22526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:13.048763 22526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "a3fb9849680e4dd7a83cbab28cf63f9c"
format_stamp: "Formatted at 2026-09-28 14:38:52 on dist-test-slave-06lh"
server_key: "282092ba3d417bf5b7d087a4af1272d5"
server_key_iv: "b85023a196d8156b04881a852cd6295c"
server_key_version: "encryptionkey@0"
I20260928 14:39:13.049325 22526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:13.065084 22526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:13.068549 22526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:13.068751 22526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:13.069360 22526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:13.070462 22526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:13.070519 22526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:13.070580 22526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:13.070608 22526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:13.079900 22526 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:45419
I20260928 14:39:13.079918 22654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:45419 every 8 connection(s)
I20260928 14:39:13.080929 22526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:39:13.081444 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 22526
I20260928 14:39:13.090165 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.082540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.6.23.254:45523)
0928 14:39:13.082679 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.082682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.082698 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:39:13.082800 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.082806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.082864 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.082962 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:13.083814 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.084400 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.085295 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.085538 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.085646 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.087042 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:13.087046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:13.087050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:13.087079 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:13.088540 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:13.088964 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:13.088970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:13.088972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:13.089052 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:13.089322 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:13.089326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:13.089328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:13.089513 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:13.089729 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:13.089869 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 14:39:13.089993 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 14:39:13 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606353, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:13.094779 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.082894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:36237 (local address 127.6.23.254:45523)
0928 14:39:13.083181 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.083186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.084295 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 14:39:13.085167 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.085170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.085222 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.085344 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:13.086612 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.087112 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.088565 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.088791 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.088903 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.091478 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:13.091482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:13.091483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:13.091505 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:13.092952 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:13.093429 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:13.093432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:13.093433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:13.093513 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:13.093807 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:13.093809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:13.093810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:13.093971 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:13.094042 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:13.094558 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 14:39:13.094656 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":61,"threads_started":1}
I20260928 14:39:13.095662 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.083417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:36237)
0928 14:39:13.084132 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:13.084183 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:13.084968 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:13.085513 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:13.085519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:13.085871 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:13.086471 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.086487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.087268 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.087273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.088440 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.088450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.088941 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.088945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.089104 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.089620 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:13.089636 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:13.091298 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:13.093055 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:13.093060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:13.093073 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:13.093320 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:13.093593 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:13.093595 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:13.093597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:13.093724 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:13.094055 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:13.094059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:13.094285 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 14:39:13.094455 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 14:39:13.094708 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":140,"threads_started":1}
I20260928 14:39:13.096715 22655 heartbeater.cc:347] Connected to a master server at 127.6.23.254:45523
I20260928 14:39:13.096961 22655 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:13.097450 22655 heartbeater.cc:511] Master 127.6.23.254:45523 requested a full tablet report, sending...
I20260928 14:39:13.098448 22197 ts_manager.cc:194] Registered new tserver with Master: a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419)
I20260928 14:39:13.098992 22197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:36237
I20260928 14:39:13.104159 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:39:13.108124 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.104793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.6.23.254:45523)
0928 14:39:13.104919 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.104922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.105115 (+ 193us) server_negotiation.cc:407] Connection header received
0928 14:39:13.105249 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.105251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.105347 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.105429 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:39:13.105643 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.106157 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.107041 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.107338 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.107419 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.107679 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:39:13.107815 (+ 136us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 14:39:13.108023 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47250: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:39:13.108211 22658 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47250: token expired
W20260928 14:39:13.108552 22128 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.23.254:45523)
I20260928 14:39:13.115622 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.108907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.6.23.254:45523)
0928 14:39:13.109030 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.109033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.109045 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:39:13.109121 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.109123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.109158 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.109213 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:13.109761 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.110223 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.110943 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.111142 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.111211 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.112685 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:13.112688 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:13.112691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:13.112713 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:13.114245 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:13.114617 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:13.114620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:13.114621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:13.114691 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:13.114907 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:13.114909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:13.114910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:13.115069 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:13.115165 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:13.115412 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 14:39:13.115506 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:39:13.118067 22197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:39:13.121443 22197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:39:13.129633 22319 tablet_service.cc:1467] Processing CreateTablet for tablet 9774f67cf8b2436998673616c7dc32e4 (DEFAULT_TABLE table=test-table [id=1005b66bb46d42b9bce9b777de3663ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:13.130330 22319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9774f67cf8b2436998673616c7dc32e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:13.133306 22277 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Bootstrap starting.
I20260928 14:39:13.134079 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.128327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:45419 (local address 127.0.0.1:35002)
0928 14:39:13.128470 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:13.128487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:13.128571 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:13.129566 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:13.129570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:13.129588 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:13.129704 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.129710 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.131249 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.131252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.132494 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.132500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.133609 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.133613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.133712 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.133760 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:39:13.133839 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:39:13.133936 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":55}
I20260928 14:39:13.134323 22277 tablet_bootstrap.cc:654] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:13.134883 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.128556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.6.23.195:45419)
0928 14:39:13.129006 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.129021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.129050 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:39:13.129134 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.129140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.129392 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.129569 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:13.129841 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.131092 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.132625 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.133501 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.133608 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.133767 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:13.134596 (+ 829us) server_negotiation.cc:299] Negotiation successful
0928 14:39:13.134711 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":154,"threads_started":1}
I20260928 14:39:13.135393 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.128595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:43483 (local address 127.0.0.1:52898)
0928 14:39:13.129176 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:13.129191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:13.129288 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:13.129778 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:13.129781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:13.129798 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:13.129966 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.129973 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.131493 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.131496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.132474 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.132487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.134871 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.134874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.135003 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.135030 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:39:13.135113 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:39:13.135206 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260928 14:39:13.135407 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.128876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.6.23.194:43483)
0928 14:39:13.129309 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.129320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.129361 (+ 41us) server_negotiation.cc:407] Connection header received
0928 14:39:13.129434 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.129440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.129639 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.129778 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:13.130093 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.131208 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.133839 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.134732 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.134846 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.135023 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:13.135144 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 14:39:13.135272 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":132,"threads_started":1}
I20260928 14:39:13.136089 22277 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: No bootstrap required, opened a new log
I20260928 14:39:13.136184 22277 ts_tablet_manager.cc:1397] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:39:13.136647 22277 raft_consensus.cc:348] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.136781 22277 raft_consensus.cc:374] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:13.136823 22277 raft_consensus.cc:729] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404165dc8c0c4073b1a25402534d3bdb, State: Initialized, Role: FOLLOWER
I20260928 14:39:13.136951 22277 consensus_queue.cc:260] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.137217 22277 ts_tablet_manager.cc:1428] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:39:13.138362 22589 tablet_service.cc:1467] Processing CreateTablet for tablet 9774f67cf8b2436998673616c7dc32e4 (DEFAULT_TABLE table=test-table [id=1005b66bb46d42b9bce9b777de3663ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:13.138387 22457 tablet_service.cc:1467] Processing CreateTablet for tablet 9774f67cf8b2436998673616c7dc32e4 (DEFAULT_TABLE table=test-table [id=1005b66bb46d42b9bce9b777de3663ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:13.139398 22589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9774f67cf8b2436998673616c7dc32e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:13.139461 22457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9774f67cf8b2436998673616c7dc32e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:13.146420 22668 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Bootstrap starting.
I20260928 14:39:13.146914 22669 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: Bootstrap starting.
I20260928 14:39:13.149003 22669 tablet_bootstrap.cc:654] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:13.149219 22668 tablet_bootstrap.cc:654] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:13.150012 22668 log.cc:824] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:13.150132 22669 log.cc:824] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:13.152184 22668 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: No bootstrap required, opened a new log
I20260928 14:39:13.152225 22669 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: No bootstrap required, opened a new log
I20260928 14:39:13.152395 22669 ts_tablet_manager.cc:1397] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:39:13.152395 22668 ts_tablet_manager.cc:1397] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:39:13.155805 22669 raft_consensus.cc:348] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.156081 22669 raft_consensus.cc:374] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:13.155934 22668 raft_consensus.cc:348] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.156169 22669 raft_consensus.cc:729] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fb9849680e4dd7a83cbab28cf63f9c, State: Initialized, Role: FOLLOWER
I20260928 14:39:13.156181 22668 raft_consensus.cc:374] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:13.156227 22668 raft_consensus.cc:729] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e415d5890284636b527334ecfeb9e9e, State: Initialized, Role: FOLLOWER
I20260928 14:39:13.156639 22669 consensus_queue.cc:260] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.156646 22668 consensus_queue.cc:260] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.157328 22655 heartbeater.cc:503] Master 127.6.23.254:45523 was elected leader, sending a full tablet report...
I20260928 14:39:13.157330 22523 heartbeater.cc:503] Master 127.6.23.254:45523 was elected leader, sending a full tablet report...
I20260928 14:39:13.157742 22669 ts_tablet_manager.cc:1428] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:39:13.158775 22668 ts_tablet_manager.cc:1428] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260928 14:39:13.188468 22386 tablet.cc:2369] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:39:13.203116 22672 raft_consensus.cc:482] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:13.203341 22672 raft_consensus.cc:504] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.204608 22672 leader_election.cc:288] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483), 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473)
I20260928 14:39:13.208940 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.204911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:43483 (local address 127.6.23.195:41917)
0928 14:39:13.205058 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:13.205074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:13.205169 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:13.205545 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:13.205547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:13.205563 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:13.205691 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.205697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.206598 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.206601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.207679 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.207688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.208482 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.208485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.208601 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.208623 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:39:13.208694 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:39:13.208769 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:39:13.209016 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.205034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:41917 (local address 127.6.23.194:43483)
0928 14:39:13.205189 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.205193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.205211 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:39:13.205273 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.205276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.205329 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.205465 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:13.205831 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.206486 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.207800 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.208352 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.208493 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.208550 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:13.208809 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 14:39:13.208884 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:39:13.209065 22393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.205321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:34261 (local address 127.6.23.193:35473)
0928 14:39:13.205473 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.205478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.205695 (+ 217us) server_negotiation.cc:407] Connection header received
0928 14:39:13.205775 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.205778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.205850 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.205925 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:13.206202 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.206852 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.207962 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.208455 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.208577 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.208733 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:13.208859 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 14:39:13.208938 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:39:13.209400 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.205191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:35473 (local address 127.6.23.195:34261)
0928 14:39:13.205607 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:13.205618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:13.205698 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:13.205969 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:13.205972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:13.205989 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:13.206095 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.206101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.206967 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.206969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.207839 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:13.207849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.208621 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.208624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:13.208726 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.208744 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:39:13.209189 (+ 445us) client_negotiation.cc:240] Negotiation successful
0928 14:39:13.209267 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":164,"threads_started":1}
I20260928 14:39:13.209664 22477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9774f67cf8b2436998673616c7dc32e4" candidate_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e415d5890284636b527334ecfeb9e9e" is_pre_election: true
I20260928 14:39:13.209966 22477 raft_consensus.cc:2454] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fb9849680e4dd7a83cbab28cf63f9c in term 0.
I20260928 14:39:13.210309 22334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9774f67cf8b2436998673616c7dc32e4" candidate_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404165dc8c0c4073b1a25402534d3bdb" is_pre_election: true
I20260928 14:39:13.210480 22543 leader_election.cc:302] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e415d5890284636b527334ecfeb9e9e, a3fb9849680e4dd7a83cbab28cf63f9c; no voters:
I20260928 14:39:13.210614 22334 raft_consensus.cc:2454] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3fb9849680e4dd7a83cbab28cf63f9c in term 0.
I20260928 14:39:13.210765 22672 raft_consensus.cc:2781] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:39:13.210868 22672 raft_consensus.cc:482] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:39:13.210945 22672 raft_consensus.cc:3037] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:13.212159 22672 raft_consensus.cc:504] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.212469 22672 leader_election.cc:288] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [CANDIDATE]: Term 1 election: Requested vote from peers 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483), 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473)
I20260928 14:39:13.212858 22477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9774f67cf8b2436998673616c7dc32e4" candidate_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e415d5890284636b527334ecfeb9e9e"
I20260928 14:39:13.212888 22334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9774f67cf8b2436998673616c7dc32e4" candidate_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404165dc8c0c4073b1a25402534d3bdb"
I20260928 14:39:13.212997 22477 raft_consensus.cc:3037] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:13.213028 22334 raft_consensus.cc:3037] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:13.214061 22477 raft_consensus.cc:2454] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fb9849680e4dd7a83cbab28cf63f9c in term 1.
I20260928 14:39:13.214072 22334 raft_consensus.cc:2454] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3fb9849680e4dd7a83cbab28cf63f9c in term 1.
I20260928 14:39:13.214376 22543 leader_election.cc:302] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e415d5890284636b527334ecfeb9e9e, a3fb9849680e4dd7a83cbab28cf63f9c; no voters:
I20260928 14:39:13.214568 22672 raft_consensus.cc:2781] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:13.214810 22672 raft_consensus.cc:686] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 LEADER]: Becoming Leader. State: Replica: a3fb9849680e4dd7a83cbab28cf63f9c, State: Running, Role: LEADER
I20260928 14:39:13.215169 22672 consensus_queue.cc:237] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:13.218344 22196 catalog_manager.cc:5495] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c reported cstate change: term changed from 0 to 1, leader changed from <none> to a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195). New cstate: current_term: 1 leader_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } health_report { overall_health: UNKNOWN } } }
I20260928 14:39:13.236099 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.232883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.6.23.195:45419)
0928 14:39:13.233023 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:13.233026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:13.233039 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:39:13.233166 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:13.233168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:13.233221 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:13.233307 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:39:13.233518 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.234015 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.234780 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:13.234991 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:13.235064 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:13.235366 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:39:13.235769 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:39:13.235859 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:13.235912 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:39:13.235970 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:39:13.242375 22477 raft_consensus.cc:1261] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 1 FOLLOWER]: Refusing update from remote peer a3fb9849680e4dd7a83cbab28cf63f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:13.242373 22334 raft_consensus.cc:1261] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Refusing update from remote peer a3fb9849680e4dd7a83cbab28cf63f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:13.243119 22675 consensus_queue.cc:1035] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:13.243360 22672 consensus_queue.cc:1035] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:13.252416 22677 mvcc.cc:205] Tried to move back new op lower bound from 7334323622872117248 to 7334323622772076544. Current Snapshot: MvccSnapshot[applied={T|T < 7334323622872117248 or (T in {7334323622872117248})}]
I20260928 14:39:13.252877 22679 mvcc.cc:205] Tried to move back new op lower bound from 7334323622872117248 to 7334323622772076544. Current Snapshot: MvccSnapshot[applied={T|T < 7334323622872117248}]
I20260928 14:39:13.259210 22196 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:13.259434 22196 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:13.261788 22196 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Sending DeleteTablet for 3 replicas of tablet 9774f67cf8b2436998673616c7dc32e4
I20260928 14:39:13.262564 22319 tablet_service.cc:1514] Processing DeleteTablet for tablet 9774f67cf8b2436998673616c7dc32e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:13 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:57746
I20260928 14:39:13.262535 22589 tablet_service.cc:1514] Processing DeleteTablet for tablet 9774f67cf8b2436998673616c7dc32e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:13 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:35002
I20260928 14:39:13.262799 22457 tablet_service.cc:1514] Processing DeleteTablet for tablet 9774f67cf8b2436998673616c7dc32e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:13 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:52898
I20260928 14:39:13.263010 22686 tablet_replica.cc:330] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: stopping tablet replica
I20260928 14:39:13.263048 22687 tablet_replica.cc:330] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: stopping tablet replica
I20260928 14:39:13.263193 22686 raft_consensus.cc:2229] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:13.263394 22686 raft_consensus.cc:2258] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:13.263505 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 22235
I20260928 14:39:13.264480 22687 raft_consensus.cc:2229] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:39:13.264611 22688 tablet_replica.cc:330] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: stopping tablet replica
I20260928 14:39:13.264840 22687 raft_consensus.cc:2258] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:13.264951 22688 raft_consensus.cc:2229] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:13.265424 22688 raft_consensus.cc:2258] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:13.266186 22687 ts_tablet_manager.cc:1905] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:13.268477 22688 ts_tablet_manager.cc:1905] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:13.269454 22687 ts_tablet_manager.cc:1918] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:13.269559 22687 log.cc:1196] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/9774f67cf8b2436998673616c7dc32e4
I20260928 14:39:13.269906 22687 ts_tablet_manager.cc:1939] T 9774f67cf8b2436998673616c7dc32e4 P a3fb9849680e4dd7a83cbab28cf63f9c: Deleting consensus metadata
W20260928 14:39:13.271283 22182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:35473 (error 108)
I20260928 14:39:13.271356 22183 catalog_manager.cc:4841] TS a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419): tablet 9774f67cf8b2436998673616c7dc32e4 (table test-table [id=1005b66bb46d42b9bce9b777de3663ec]) successfully deleted
I20260928 14:39:13.271355 22688 ts_tablet_manager.cc:1918] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:13.271510 22688 log.cc:1196] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/9774f67cf8b2436998673616c7dc32e4
I20260928 14:39:13.271517 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 22389
I20260928 14:39:13.277776 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.277380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:35473 (local address 127.0.0.1:57758)
0928 14:39:13.277566 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:13.277661 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:35473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 14:39:13.278038 22182 catalog_manager.cc:4584] TS 404165dc8c0c4073b1a25402534d3bdb (127.6.23.193:35473): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9774f67cf8b2436998673616c7dc32e4: Network error: Client connection negotiation failed: client connection to 127.6.23.193:35473: connect: Connection refused (error 111)
I20260928 14:39:13.278733 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 22526
I20260928 14:39:13.280331 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:13.280071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:43483 (local address 127.0.0.1:52910)
0928 14:39:13.280198 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:13.280254 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.194:43483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:39:13.284898 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 22164
I20260928 14:39:33.291643 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45523
--webserver_interface=127.6.23.254
--webserver_port=44901
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:33.412863 22690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:33.413174 22690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:33.413218 22690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:33.417392 22690 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:39:33.417465 22690 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:39:33.417481 22690 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:39:33.417495 22690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:39:33.417515 22690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:33.417533 22690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:33.417550 22690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:39:33.417567 22690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:39:33.423401 22690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:33.424531 22690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:33.425561 22690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:33.435194 22698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:33.435298 22690 server_base.cc:1034] running on GCE node
W20260928 14:39:33.435205 22696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:33.435205 22695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:33.436156 22690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:33.437368 22690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:33.438575 22690 hybrid_clock.cc:648] HybridClock initialized: now 1790606373438556 us; error 34 us; skew 500 ppm
Sep 28 14:39:33 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606373, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:33.448982 22690 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:39:33.450084 22690 webserver.cc:460] Webserver started at http://127.6.23.254:44901/ using document root <none> and password file <none>
I20260928 14:39:33.450678 22690 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:33.450743 22690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:33.455052 22690 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:39:33.457535 22705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:33.458945 22690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:39:33.459065 22690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "78c9cf0225e87744a7adc3976292331b"
server_key_iv: "fb10705edef90937b4572a1ee7f1d779"
server_key_version: "encryptionkey@0"
I20260928 14:39:33.459728 22690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:33.489197 22690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:33.492767 22690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:33.492942 22690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:33.500561 22690 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:45523
I20260928 14:39:33.500859 22757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:45523 every 8 connection(s)
I20260928 14:39:33.501770 22690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:39:33.506202 22758 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:39:33.508095 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Bootstrap starting.
I20260928 14:39:33.509452 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 22690
I20260928 14:39:33.510102 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:35473
--local_ip_for_outbound_sockets=127.6.23.193
--tserver_master_addrs=127.6.23.254:45523
--webserver_port=34953
--webserver_interface=127.6.23.193
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:39:33.527853 22758 log.cc:824] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:33.539321 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:39:33.539779 22758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Bootstrap complete.
I20260928 14:39:33.543957 22758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:39:33.544394 22758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f0624b724c4bb1ac9af0c13d8ff8b1, State: Initialized, Role: FOLLOWER
I20260928 14:39:33.544937 22758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:39:33.545120 22758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:39:33.545199 22758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:39:33.545317 22758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:39:33.547449 22758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:39:33.547807 22758 leader_election.cc:302] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16f0624b724c4bb1ac9af0c13d8ff8b1; no voters:
I20260928 14:39:33.548146 22758 leader_election.cc:288] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:39:33.548255 22762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:39:33.548460 22762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [term 3 LEADER]: Becoming Leader. State: Replica: 16f0624b724c4bb1ac9af0c13d8ff8b1, State: Running, Role: LEADER
I20260928 14:39:33.548785 22762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } }
I20260928 14:39:33.549489 22758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:39:33.549824 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16f0624b724c4bb1ac9af0c13d8ff8b1. Latest consensus state: current_term: 3 leader_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } } }
I20260928 14:39:33.549958 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:33.550175 22763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f0624b724c4bb1ac9af0c13d8ff8b1" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45523 } } }
I20260928 14:39:33.550293 22763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:33.550624 22766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:39:33.556411 22766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:39:33.557313 22766 catalog_manager.cc:1259] Loaded cluster ID: 4b77008b3a16454bafc28bd49dcdbccc
I20260928 14:39:33.557390 22766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:39:33.560230 22766 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:39:33.561076 22766 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Loaded TSK: 0
I20260928 14:39:33.561177 22766 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16f0624b724c4bb1ac9af0c13d8ff8b1: Loaded TSK: 1
F20260928 14:39:33.562057 22766 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8be6bcd9ad google::LogMessage::Fail() at ??:0
@ 0x7f8be6bd088a google::LogMessage::SendToLog() at ??:0
@ 0x7f8be6bcd4de google::LogMessage::Flush() at ??:0
@ 0x7f8be6bcea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8be989118d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8be9889f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8bf15367bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8bf151099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8bf1511178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f8bf1552df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8bf157182b std::function<>::operator()() at ??:0
@ 0x7f8bf1510b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8bf1511f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8bf150e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8bf1552640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8be8e1f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8be8e1fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8be8e213f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8be8e10512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8bea4546db start_thread at ??:0
@ 0x7f8be56cb61f clone at ??:0
W20260928 14:39:33.660185 22760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:33.660619 22760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:33.660691 22760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:33.666980 22760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:33.667066 22760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:33.667160 22760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:39:33.675572 22760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:35473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=34953
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:33.677030 22760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:33.678476 22760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:39:33.690601 22760 server_base.cc:1034] running on GCE node
W20260928 14:39:33.690613 22805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:33.690524 22806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:33.690524 22808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:33.691290 22760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:33.691944 22760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:33.693132 22760 hybrid_clock.cc:648] HybridClock initialized: now 1790606373693108 us; error 45 us; skew 500 ppm
Sep 28 14:39:33 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606373, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:33.696053 22760 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:39:33.696998 22760 webserver.cc:460] Webserver started at http://127.6.23.193:34953/ using document root <none> and password file <none>
I20260928 14:39:33.697577 22760 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:33.697682 22760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:33.701743 22760 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:39:33.704120 22815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:33.705461 22760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:39:33.705603 22760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "404165dc8c0c4073b1a25402534d3bdb"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "b3eaa67dd960254738a51807e4d4dcef"
server_key_iv: "91a3d2efa1cf6b2729088282907557ac"
server_key_version: "encryptionkey@0"
I20260928 14:39:33.706173 22760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:33.722711 22760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:33.726130 22760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:33.726353 22760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:33.726971 22760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:33.728223 22822 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:39:33.733774 22760 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:39:33.733880 22760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 14:39:33.733975 22760 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:39:33.736212 22760 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:39:33.736287 22760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:33.736583 22822 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Bootstrap starting.
I20260928 14:39:33.747102 22760 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:35473
I20260928 14:39:33.747318 22929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:35473 every 8 connection(s)
I20260928 14:39:33.748219 22760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:39:33.748625 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 22760
I20260928 14:39:33.749104 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:43483
--local_ip_for_outbound_sockets=127.6.23.194
--tserver_master_addrs=127.6.23.254:45523
--webserver_port=35299
--webserver_interface=127.6.23.194
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:39:33.758265 22822 log.cc:824] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:33.765314 22822 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:39:33.765740 22822 tablet_bootstrap.cc:492] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Bootstrap complete.
I20260928 14:39:33.767000 22822 ts_tablet_manager.cc:1397] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.000s
I20260928 14:39:33.769481 22822 raft_consensus.cc:348] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:33.769920 22822 raft_consensus.cc:729] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404165dc8c0c4073b1a25402534d3bdb, State: Initialized, Role: FOLLOWER
I20260928 14:39:33.770407 22822 consensus_queue.cc:260] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:33.771426 22822 ts_tablet_manager.cc:1428] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb: Time spent starting tablet: real 0.004s user 0.008s sys 0.000s
W20260928 14:39:33.829366 22933 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:39:33.750619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:45769)
0928 14:39:33.751935 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:33.751999 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:33.753139 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:33.828306 (+ 75167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":168,"threads_started":1}
W20260928 14:39:33.829943 22930 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:45523 (0 consecutive failures): Network error: Failed to ping master at 127.6.23.254:45523: Client connection negotiation failed: client connection to 127.6.23.254:45523: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:39:33.830790 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:33.830491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:33789)
0928 14:39:33.830643 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:33.830723 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:39:33.831597 22933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:33.831375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:58899)
0928 14:39:33.831489 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:33.831540 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:39:33.831811 22930 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:39:33.885337 22932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:33.885615 22932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:33.885660 22932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:33.889780 22932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:33.889840 22932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:33.889937 22932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:39:33.895246 22932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:43483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=35299
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:33.896315 22932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:33.897395 22932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:33.907196 22940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:33.907277 22941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:33.907415 22932 server_base.cc:1034] running on GCE node
W20260928 14:39:33.907195 22943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:33.908154 22932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:33.908756 22932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:33.909919 22932 hybrid_clock.cc:648] HybridClock initialized: now 1790606373909889 us; error 49 us; skew 500 ppm
Sep 28 14:39:33 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606373, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:33.912781 22932 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:39:33.913669 22932 webserver.cc:460] Webserver started at http://127.6.23.194:35299/ using document root <none> and password file <none>
I20260928 14:39:33.914201 22932 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:33.914356 22932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:33.918395 22932 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:33.922580 22950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:33.924185 22932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:39:33.924347 22932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "1e415d5890284636b527334ecfeb9e9e"
format_stamp: "Formatted at 2026-09-28 14:38:51 on dist-test-slave-06lh"
server_key: "a9f302fe40a09f2b8a5833f3ae27330e"
server_key_iv: "4a8b14ca5b95b68a4bfd2cdff409fdc1"
server_key_version: "encryptionkey@0"
I20260928 14:39:33.924980 22932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:33.940325 22932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:33.943859 22932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:33.944082 22932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:33.944711 22932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:33.945814 22957 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 14:39:33.949750 22957 ts_tablet_manager.cc:1852] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 14:39:33.950138 22957 ts_tablet_manager.cc:1905] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:33.954293 22957 ts_tablet_manager.cc:1918] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:33.954490 22957 ts_tablet_manager.cc:1939] T 9774f67cf8b2436998673616c7dc32e4 P 1e415d5890284636b527334ecfeb9e9e: Deleting consensus metadata
I20260928 14:39:33.954882 22932 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 14:39:33.954979 22932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260928 14:39:33.955070 22932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:33.955116 22932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:33.964715 22932 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:43483
I20260928 14:39:33.964747 23064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:43483 every 8 connection(s)
I20260928 14:39:33.965637 22932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:39:33.969007 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 22932
I20260928 14:39:33.969298 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:33.967892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:33127)
0928 14:39:33.968416 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:33.968529 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":122,"threads_started":1}
I20260928 14:39:33.969507 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:45419
--local_ip_for_outbound_sockets=127.6.23.195
--tserver_master_addrs=127.6.23.254:45523
--webserver_port=32931
--webserver_interface=127.6.23.195
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:33.970966 23065 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:45523 (0 consecutive failures): Network error: Failed to ping master at 127.6.23.254:45523: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
I20260928 14:39:33.971711 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:33.971412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:47995)
0928 14:39:33.971548 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:33.971618 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 14:39:33.972424 23067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:33.972203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:41709)
0928 14:39:33.972322 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:33.972374 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 14:39:33.972657 23065 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:39:34.083060 23068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:34.083422 23068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:34.083504 23068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:34.087585 23068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:34.087643 23068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:34.087738 23068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:39:34.093024 23068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:45419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=32931
--tserver_master_addrs=127.6.23.254:45523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:34.094112 23068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:34.095173 23068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:34.104811 23074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:34.104846 23073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:34.104846 23076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:34.105018 23068 server_base.cc:1034] running on GCE node
I20260928 14:39:34.105499 23068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:34.106133 23068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:34.107329 23068 hybrid_clock.cc:648] HybridClock initialized: now 1790606374107294 us; error 49 us; skew 500 ppm
Sep 28 14:39:34 dist-test-slave-06lh krb5kdc[21618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606374, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:34.110109 23068 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:39:34.111107 23068 webserver.cc:460] Webserver started at http://127.6.23.195:32931/ using document root <none> and password file <none>
I20260928 14:39:34.111706 23068 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:34.111770 23068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:34.115696 23068 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:39:34.117859 23083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:34.119208 23068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:34.119414 23068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "a3fb9849680e4dd7a83cbab28cf63f9c"
format_stamp: "Formatted at 2026-09-28 14:38:52 on dist-test-slave-06lh"
server_key: "282092ba3d417bf5b7d087a4af1272d5"
server_key_iv: "b85023a196d8156b04881a852cd6295c"
server_key_version: "encryptionkey@0"
I20260928 14:39:34.119997 23068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:34.146456 23068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:34.150152 23068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:34.150393 23068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:34.151034 23068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:34.151942 23068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:34.152012 23068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:34.152084 23068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:34.152127 23068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:34.161548 23068 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:45419
I20260928 14:39:34.161566 23196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:45419 every 8 connection(s)
I20260928 14:39:34.162597 23068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:39:34.166335 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:34.164791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:49013)
0928 14:39:34.165482 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:34.165596 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":118,"threads_started":1}
W20260928 14:39:34.166808 23197 heartbeater.cc:644] Failed to heartbeat to 127.6.23.254:45523 (0 consecutive failures): Network error: Failed to ping master at 127.6.23.254:45523: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
I20260928 14:39:34.166832 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 23068
I20260928 14:39:34.167562 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:34.167315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:59077)
0928 14:39:34.167452 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:34.167510 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:39:34.168331 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:34.168125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:50023)
0928 14:39:34.168234 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:34.168284 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:39:34.168536 23197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:39:34.833065 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:34.832530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:37885)
0928 14:39:34.832845 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:34.832931 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
I20260928 14:39:34.973737 23202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:34.973310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:57707)
0928 14:39:34.973560 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:34.973621 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
I20260928 14:39:35.119068 23203 raft_consensus.cc:482] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:35.119366 23203 raft_consensus.cc:504] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:35.120574 23203 leader_election.cc:288] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419), 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483)
Sep 28 14:39:35 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606373, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.194@KRBTEST.COM
Sep 28 14:39:35 dist-test-slave-06lh krb5kdc[21618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606373, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.195@KRBTEST.COM
I20260928 14:39:35.138935 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:35.120781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:45419 (local address 127.6.23.193:40185)
0928 14:39:35.121008 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:35.121023 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:35.121133 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:35.123436 (+ 2303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:35.123439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:35.124098 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:35.124992 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:35.125014 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:35.126998 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:35.127002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:35.128799 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:35.128813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:35.129650 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:35.129654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:35.129896 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:35.130632 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:35.130649 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:35.132813 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:35.135350 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:35.135357 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:35.135375 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:35.135867 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:35.136396 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:35.136400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:35.136402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:35.136683 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:35.137610 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:35.137618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:35.138040 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 14:39:35.138284 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 14:39:35.138720 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":437,"spinlock_wait_cycles":57216}
I20260928 14:39:35.138991 23206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:35.121160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:40185 (local address 127.6.23.195:45419)
0928 14:39:35.121576 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:35.121586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:35.121614 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:39:35.122382 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:35.122391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:35.122893 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:35.123272 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:35.125201 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:35.126800 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:35.128973 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:35.129465 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:35.129707 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:35.133016 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:35.133024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:35.133041 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:35.133078 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:35.135140 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:35.136054 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:35.136060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:35.136067 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:35.136258 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:35.136824 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:35.136827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:35.136829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:35.137463 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:35.137603 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:35.138485 (+ 882us) server_negotiation.cc:299] Negotiation successful
0928 14:39:35.138800 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":121,"threads_started":1}
I20260928 14:39:35.138993 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:35.121007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:58571 (local address 127.6.23.194:43483)
0928 14:39:35.121438 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:35.121455 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:35.121500 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:39:35.122488 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:35.122495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:35.122891 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:35.123272 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:35.125207 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:35.126800 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:35.128961 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:35.129468 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:35.129707 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:35.132800 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:35.132805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:35.132819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:35.132849 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:35.135140 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:35.136054 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:35.136060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:35.136067 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:35.136258 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:35.136824 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:35.136827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:35.136829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:35.137456 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:35.137603 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:35.138482 (+ 879us) server_negotiation.cc:299] Negotiation successful
0928 14:39:35.138800 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":80,"threads_started":1}
I20260928 14:39:35.138954 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:35.120820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:43483 (local address 127.6.23.193:58571)
0928 14:39:35.121220 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:35.121235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:35.121378 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:35.123424 (+ 2046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:35.123432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:35.124098 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:35.124992 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:35.125014 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:35.126998 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:35.127002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:35.128813 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:35.128820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:35.129650 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:35.129653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:35.129893 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:35.130671 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:35.130683 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:35.132549 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:35.135350 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:35.135357 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:35.135375 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:35.135867 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:35.136396 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:35.136400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:35.136402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:35.136683 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:35.137610 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:35.137618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:35.138040 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 14:39:35.138284 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 14:39:35.138773 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":598,"spinlock_wait_cycles":27008,"thread_start_us":84,"threads_started":1}
W20260928 14:39:35.142277 22817 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
W20260928 14:39:35.142443 22818 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
I20260928 14:39:35.142503 22817 leader_election.cc:302] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 404165dc8c0c4073b1a25402534d3bdb; no voters: 1e415d5890284636b527334ecfeb9e9e, a3fb9849680e4dd7a83cbab28cf63f9c
I20260928 14:39:35.142805 23203 raft_consensus.cc:2726] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:39:35.169595 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:35.169164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:35859)
0928 14:39:35.169452 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:35.169503 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":98,"threads_started":1}
I20260928 14:39:35.834281 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:35.833851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:43405)
0928 14:39:35.834124 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:35.834184 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260928 14:39:35.975040 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:35.974594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:51935)
0928 14:39:35.974886 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:35.974946 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260928 14:39:36.170924 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:36.170404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:52545)
0928 14:39:36.170757 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:36.170813 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
I20260928 14:39:36.835609 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:36.835126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:50389)
0928 14:39:36.835445 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:36.835508 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":130,"threads_started":1}
I20260928 14:39:36.968196 23212 raft_consensus.cc:482] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:36.968356 23212 raft_consensus.cc:504] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:36.968811 23212 leader_election.cc:288] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419), 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483)
W20260928 14:39:36.969679 22818 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
W20260928 14:39:36.969774 22817 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
I20260928 14:39:36.969882 22817 leader_election.cc:302] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 404165dc8c0c4073b1a25402534d3bdb; no voters: 1e415d5890284636b527334ecfeb9e9e, a3fb9849680e4dd7a83cbab28cf63f9c
I20260928 14:39:36.970034 23212 raft_consensus.cc:2726] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:39:36.976158 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:36.975786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:49055)
0928 14:39:36.976018 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:36.976070 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":68,"threads_started":1}
I20260928 14:39:37.172235 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:37.171783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:55007)
0928 14:39:37.172079 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:37.172141 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":100,"threads_started":1}
I20260928 14:39:37.836942 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:37.836498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:34783)
0928 14:39:37.836781 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:37.836845 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260928 14:39:37.977418 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:37.976923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:55683)
0928 14:39:37.977219 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:37.977308 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
I20260928 14:39:38.173472 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:38.173059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:42037)
0928 14:39:38.173321 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:38.173378 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260928 14:39:38.838461 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:38.837960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:58767)
0928 14:39:38.838297 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:38.838363 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":78,"threads_started":1}
I20260928 14:39:38.978870 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:38.978390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:51495)
0928 14:39:38.978654 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:38.978754 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260928 14:39:39.174741 23220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:39.174320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:39013)
0928 14:39:39.174570 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:39.174634 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260928 14:39:39.839813 23221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:39.839376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:40239)
0928 14:39:39.839641 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:39.839706 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260928 14:39:39.961726 23222 raft_consensus.cc:482] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:39.961869 23222 raft_consensus.cc:504] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:39.962363 23222 leader_election.cc:288] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419), 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483)
W20260928 14:39:39.963163 22817 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
W20260928 14:39:39.963302 22818 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
I20260928 14:39:39.963377 22817 leader_election.cc:302] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 404165dc8c0c4073b1a25402534d3bdb; no voters: 1e415d5890284636b527334ecfeb9e9e, a3fb9849680e4dd7a83cbab28cf63f9c
I20260928 14:39:39.963538 23222 raft_consensus.cc:2726] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:39:39.980279 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:39.979727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:38307)
0928 14:39:39.980079 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:39.980160 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260928 14:39:40.176009 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:40.175565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:55001)
0928 14:39:40.175847 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:40.175910 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
I20260928 14:39:40.841158 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:40.840717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:56603)
0928 14:39:40.840989 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:40.841055 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":77,"threads_started":1}
I20260928 14:39:40.981534 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:40.981068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:39195)
0928 14:39:40.981371 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:40.981434 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":74,"threads_started":1}
I20260928 14:39:41.177443 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:41.176935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:46101)
0928 14:39:41.177260 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:41.177333 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":77,"threads_started":1}
I20260928 14:39:41.842664 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:41.842216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:59587)
0928 14:39:41.842499 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:41.842564 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
I20260928 14:39:41.982807 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:41.982364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:33117)
0928 14:39:41.982646 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:41.982708 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260928 14:39:42.178685 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:42.178277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:41453)
0928 14:39:42.178527 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:42.178588 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260928 14:39:42.843979 23231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:42.843519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:34589)
0928 14:39:42.843806 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:42.843869 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260928 14:39:42.984149 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:42.983679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:60003)
0928 14:39:42.983972 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:42.984037 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260928 14:39:43.180042 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:43.179597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:36171)
0928 14:39:43.179899 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:43.179948 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":120,"threads_started":1}
I20260928 14:39:43.214706 23234 raft_consensus.cc:482] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:43.214843 23234 raft_consensus.cc:504] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:43.215302 23234 leader_election.cc:288] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419), 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483)
W20260928 14:39:43.216053 22818 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
W20260928 14:39:43.216141 22817 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
I20260928 14:39:43.216264 22817 leader_election.cc:302] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 404165dc8c0c4073b1a25402534d3bdb; no voters: 1e415d5890284636b527334ecfeb9e9e, a3fb9849680e4dd7a83cbab28cf63f9c
I20260928 14:39:43.216413 23234 raft_consensus.cc:2726] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:39:43.845319 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:43.844901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:41563)
0928 14:39:43.845154 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:43.845213 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260928 14:39:43.985412 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:43.984976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:50655)
0928 14:39:43.985252 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:43.985316 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260928 14:39:44.181301 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:44.180858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:55469)
0928 14:39:44.181109 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:44.181167 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260928 14:39:44.846621 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:44.846152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:50277)
0928 14:39:44.846450 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:44.846511 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260928 14:39:44.986652 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:44.986205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:34267)
0928 14:39:44.986495 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:44.986555 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260928 14:39:45.182698 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:45.182207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:54955)
0928 14:39:45.182518 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:45.182578 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":128,"threads_started":1}
I20260928 14:39:45.847972 23241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:45.847516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:54353)
0928 14:39:45.847814 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:45.847876 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260928 14:39:45.987898 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:45.987476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:39705)
0928 14:39:45.987734 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:45.987788 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260928 14:39:46.184213 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:46.183686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:52985)
0928 14:39:46.184033 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:46.184104 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":129,"threads_started":1}
I20260928 14:39:46.849315 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:46.848832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:37049)
0928 14:39:46.849145 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:46.849209 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260928 14:39:46.968161 23245 raft_consensus.cc:482] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:46.968304 23245 raft_consensus.cc:504] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3fb9849680e4dd7a83cbab28cf63f9c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 45419 } } peers { permanent_uuid: "1e415d5890284636b527334ecfeb9e9e" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 43483 } } peers { permanent_uuid: "404165dc8c0c4073b1a25402534d3bdb" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 35473 } }
I20260928 14:39:46.968719 23245 leader_election.cc:288] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419), 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483)
W20260928 14:39:46.969547 22817 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1e415d5890284636b527334ecfeb9e9e (127.6.23.194:43483): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
W20260928 14:39:46.969659 22818 leader_election.cc:341] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3fb9849680e4dd7a83cbab28cf63f9c (127.6.23.195:45419): Not found: Tablet not found: 9774f67cf8b2436998673616c7dc32e4
I20260928 14:39:46.969715 22818 leader_election.cc:302] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 404165dc8c0c4073b1a25402534d3bdb; no voters: 1e415d5890284636b527334ecfeb9e9e, a3fb9849680e4dd7a83cbab28cf63f9c
I20260928 14:39:46.969857 23245 raft_consensus.cc:2726] T 9774f67cf8b2436998673616c7dc32e4 P 404165dc8c0c4073b1a25402534d3bdb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:39:46.989109 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:46.988683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:43849)
0928 14:39:46.988954 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:46.989013 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
I20260928 14:39:47.185402 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:47.184972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:33045)
0928 14:39:47.185238 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:47.185305 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260928 14:39:47.850653 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:47.850149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:37281)
0928 14:39:47.850480 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:47.850546 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":110,"threads_started":1}
I20260928 14:39:47.990507 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:47.989971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:43361)
0928 14:39:47.990319 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:47.990404 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260928 14:39:48.186842 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:48.186355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.195:59385)
0928 14:39:48.186674 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:48.186743 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":102,"threads_started":1}
I20260928 14:39:48.852347 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:48.851811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.193:57041)
0928 14:39:48.852150 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:48.852219 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":80,"threads_started":1}
I20260928 14:39:48.992020 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:48.991522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45523 (local address 127.6.23.194:57143)
0928 14:39:48.991850 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:48.991920 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.254:45523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":76,"threads_started":1}
I20260928 14:39:49.171072 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 22760
I20260928 14:39:49.177300 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 22932
I20260928 14:39:49.182961 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 23068
2026-09-28T14:39:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59210 ms)
[----------] 30 tests from SecurityITest (246368 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:39:49 dist-test-slave-06lh krb5kdc[23255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:39:49 dist-test-slave-06lh krb5kdc[23255](info): set up 2 sockets
Sep 28 14:39:49 dist-test-slave-06lh krb5kdc[23255](info): commencing operation
krb5kdc: starting...
W20260928 14:39:50.223428 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:39:50 dist-test-slave-06lh krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:39:50.349786 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:34495
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:34495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:50.463778 23268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:50.464111 23268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:50.464208 23268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:50.468391 23268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:39:50.468484 23268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:50.468523 23268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:50.468573 23268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:39:50.468607 23268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:39:50.474162 23268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:34495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:34495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:50.475353 23268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:50.476413 23268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:50.485522 23276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:50.485595 23268 server_base.cc:1034] running on GCE node
W20260928 14:39:50.485497 23273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:50.485502 23274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:50.486359 23268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:50.487602 23268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:50.488804 23268 hybrid_clock.cc:648] HybridClock initialized: now 1790606390488786 us; error 32 us; skew 500 ppm
Sep 28 14:39:50 dist-test-slave-06lh krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:50.491803 23268 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:39:50.492781 23268 webserver.cc:460] Webserver started at http://127.6.23.254:37289/ using document root <none> and password file <none>
I20260928 14:39:50.493381 23268 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:50.493471 23268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:50.493700 23268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:50.495949 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "f71d559556d146c18bb7fb0b82110d87"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "fec7836a7e99d4456c0894bb867c13d1"
server_key_iv: "5549acc2f6161f31393cde3158c870b9"
server_key_version: "encryptionkey@0"
I20260928 14:39:50.496558 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "f71d559556d146c18bb7fb0b82110d87"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "fec7836a7e99d4456c0894bb867c13d1"
server_key_iv: "5549acc2f6161f31393cde3158c870b9"
server_key_version: "encryptionkey@0"
I20260928 14:39:50.500382 23268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 14:39:50.502926 23283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:50.504331 23268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:39:50.504482 23268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "f71d559556d146c18bb7fb0b82110d87"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "fec7836a7e99d4456c0894bb867c13d1"
server_key_iv: "5549acc2f6161f31393cde3158c870b9"
server_key_version: "encryptionkey@0"
I20260928 14:39:50.504603 23268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:50.529246 23268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:50.532829 23268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:50.533048 23268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:50.540728 23268 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:34495
I20260928 14:39:50.540758 23335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:34495 every 8 connection(s)
I20260928 14:39:50.541746 23268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:39:50.544683 23336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:50.546803 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 23268
I20260928 14:39:50.546912 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:39:50.547297 6239 external_mini_cluster.cc:1428] Setting key d4eda94054b3fe6f4622be91ac5639fb
I20260928 14:39:50.552024 23336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87: Bootstrap starting.
I20260928 14:39:50.554836 23336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:50.555871 23336 log.cc:824] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:39:50 dist-test-slave-06lh krb5kdc[23255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:50.558115 23336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87: No bootstrap required, opened a new log
I20260928 14:39:50.561748 23336 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d559556d146c18bb7fb0b82110d87" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34495 } }
I20260928 14:39:50.562034 23336 raft_consensus.cc:374] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:50.562110 23336 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71d559556d146c18bb7fb0b82110d87, State: Initialized, Role: FOLLOWER
I20260928 14:39:50.562566 23336 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d559556d146c18bb7fb0b82110d87" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34495 } }
I20260928 14:39:50.562714 23336 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:39:50.562796 23336 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:39:50.562819 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:50.548773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.6.23.254:34495)
0928 14:39:50.549219 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:50.549238 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:50.549288 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:39:50.549991 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:50.550009 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:50.550369 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:50.550680 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:50.551707 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:50.552848 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:50.554101 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:50.554613 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:50.554761 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:50.557711 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:50.557717 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:50.557732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:50.557761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:50.559701 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:50.560219 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:50.560224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:50.560229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:50.560343 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:50.560618 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:50.560622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:50.560624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:50.561037 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:50.561258 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:50.561521 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 14:39:50.561824 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":151,"threads_started":1}
I20260928 14:39:50.562897 23336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:50.564137 23336 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d559556d146c18bb7fb0b82110d87" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34495 } }
I20260928 14:39:50.564484 23336 leader_election.cc:302] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71d559556d146c18bb7fb0b82110d87; no voters:
I20260928 14:39:50.564826 23336 leader_election.cc:288] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:39:50.564934 23341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:50.565435 23341 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [term 1 LEADER]: Becoming Leader. State: Replica: f71d559556d146c18bb7fb0b82110d87, State: Running, Role: LEADER
I20260928 14:39:50.565810 23341 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d559556d146c18bb7fb0b82110d87" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34495 } }
I20260928 14:39:50.566164 23336 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:39:50.567585 23343 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71d559556d146c18bb7fb0b82110d87. Latest consensus state: current_term: 1 leader_uuid: "f71d559556d146c18bb7fb0b82110d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d559556d146c18bb7fb0b82110d87" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34495 } } }
I20260928 14:39:50.567736 23343 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:50.568044 23342 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f71d559556d146c18bb7fb0b82110d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d559556d146c18bb7fb0b82110d87" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34495 } } }
I20260928 14:39:50.568128 23342 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:50.568153 23352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:39:50.571408 23352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:39:50.576967 23352 catalog_manager.cc:1347] Generated new cluster ID: 338eaab8557e48ddaeeefed85b6fdb4a
I20260928 14:39:50.577051 23352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:39:50.586241 23352 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:39:50.587342 23352 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:39:50.596038 23352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87: Generated new TSK 0
I20260928 14:39:50.596851 23352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:39:50.632345 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34495
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:50.751739 23362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:50.752034 23362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:50.752117 23362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:50.756419 23362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:50.756500 23362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:50.756604 23362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:39:50.762223 23362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:50.763401 23362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:50.764482 23362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:50.774858 23367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:50.774971 23362 server_base.cc:1034] running on GCE node
W20260928 14:39:50.774863 23370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:50.774858 23368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:50.775626 23362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:50.776244 23362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:50.777402 23362 hybrid_clock.cc:648] HybridClock initialized: now 1790606390777383 us; error 32 us; skew 500 ppm
Sep 28 14:39:50 dist-test-slave-06lh krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:50.780207 23362 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:39:50.781113 23362 webserver.cc:460] Webserver started at http://127.6.23.193:43055/ using document root <none> and password file <none>
I20260928 14:39:50.781718 23362 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:50.781808 23362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:50.782047 23362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:50.784613 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "8d0ee8009c13443fb2ab33e93a164384"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "62c84213c765c3c745c02f52e783a669"
server_key_iv: "af7deaa8c4c615f9916a95e392f31152"
server_key_version: "encryptionkey@0"
I20260928 14:39:50.785197 23362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "8d0ee8009c13443fb2ab33e93a164384"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "62c84213c765c3c745c02f52e783a669"
server_key_iv: "af7deaa8c4c615f9916a95e392f31152"
server_key_version: "encryptionkey@0"
I20260928 14:39:50.789069 23362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 14:39:50.791543 23377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:50.792785 23362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:39:50.792933 23362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "8d0ee8009c13443fb2ab33e93a164384"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "62c84213c765c3c745c02f52e783a669"
server_key_iv: "af7deaa8c4c615f9916a95e392f31152"
server_key_version: "encryptionkey@0"
I20260928 14:39:50.793051 23362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:50.804330 23362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:50.807605 23362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:50.807809 23362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:50.808490 23362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:50.809473 23362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:50.809540 23362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:50.809612 23362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:50.809651 23362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:50.818925 23362 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:41753
I20260928 14:39:50.818943 23490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:41753 every 8 connection(s)
I20260928 14:39:50.820052 23362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
Sep 28 14:39:50 dist-test-slave-06lh krb5kdc[23255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:50.829430 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 23362
I20260928 14:39:50.829573 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:39:50.829900 6239 external_mini_cluster.cc:1428] Setting key 48e26839ed4fe9ed6fea0578cda98c43
I20260928 14:39:50.833669 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:50.821906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:32949 (local address 127.6.23.254:34495)
0928 14:39:50.822054 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:50.822058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:50.822918 (+ 860us) server_negotiation.cc:407] Connection header received
0928 14:39:50.823901 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:50.823904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:50.823963 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:50.824064 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:50.825257 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:50.825798 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:50.826950 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:50.827208 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:50.827359 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:50.829893 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:50.829898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:50.829901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:50.829934 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:50.831505 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:50.832008 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:50.832014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:50.832018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:50.832112 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:50.832448 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:50.832454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:50.832458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:50.832719 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:50.832818 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:50.833369 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 14:39:50.833505 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:39:50.834649 23493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:50.822303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34495 (local address 127.6.23.193:32949)
0928 14:39:50.822763 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:50.822807 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:50.823685 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:50.824239 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:50.824246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:50.824630 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:50.825113 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:50.825127 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:50.825935 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:50.825939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:50.826827 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:50.826836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:50.827408 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:50.827413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:50.827584 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:50.828149 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:50.828165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:50.829709 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:50.831616 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:50.831622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:50.831636 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:50.831912 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:50.832211 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:50.832213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:50.832214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:50.832363 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:50.832831 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:50.832836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:50.833120 (+ 284us) client_negotiation.cc:769] Sending connection context
0928 14:39:50.833318 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 14:39:50.833670 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
I20260928 14:39:50.835958 23491 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34495
I20260928 14:39:50.836246 23491 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:50.836925 23491 heartbeater.cc:511] Master 127.6.23.254:34495 requested a full tablet report, sending...
I20260928 14:39:50.838588 23300 ts_manager.cc:194] Registered new tserver with Master: 8d0ee8009c13443fb2ab33e93a164384 (127.6.23.193:41753)
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:39:50.839859 23300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:32949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:39:50.856487 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34495
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:50.969383 23496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:50.969669 23496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:50.969713 23496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:50.973793 23496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:50.973861 23496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:50.973922 23496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:39:50.979177 23496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:50.980316 23496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:50.981338 23496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:50.991003 23502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:50.991027 23504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:50.991035 23501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:50.991338 23496 server_base.cc:1034] running on GCE node
I20260928 14:39:50.991762 23496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:50.992429 23496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:50.993597 23496 hybrid_clock.cc:648] HybridClock initialized: now 1790606390993551 us; error 49 us; skew 500 ppm
Sep 28 14:39:50 dist-test-slave-06lh krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:50.996452 23496 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:39:50.997411 23496 webserver.cc:460] Webserver started at http://127.6.23.194:38655/ using document root <none> and password file <none>
I20260928 14:39:50.997972 23496 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:50.998034 23496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:50.998270 23496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:51.000363 23496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "8b9a8a51b8be44708eda946b3821b733"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "97204bd411b0d6221555f2c45ee7bae5"
server_key_iv: "7973fe521cf2a2605585ffab7105b7f6"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.000923 23496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "8b9a8a51b8be44708eda946b3821b733"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "97204bd411b0d6221555f2c45ee7bae5"
server_key_iv: "7973fe521cf2a2605585ffab7105b7f6"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.004673 23496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:39:51.007105 23511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.008446 23496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:39:51.008575 23496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "8b9a8a51b8be44708eda946b3821b733"
format_stamp: "Formatted at 2026-09-28 14:39:50 on dist-test-slave-06lh"
server_key: "97204bd411b0d6221555f2c45ee7bae5"
server_key_iv: "7973fe521cf2a2605585ffab7105b7f6"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.008689 23496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:51.018097 23496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:51.021385 23496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:51.021590 23496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:51.022167 23496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:51.022989 23496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:51.023056 23496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.023135 23496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:51.023173 23496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.032260 23496 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:44227
I20260928 14:39:51.032279 23624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:44227 every 8 connection(s)
I20260928 14:39:51.033175 23496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
Sep 28 14:39:51 dist-test-slave-06lh krb5kdc[23255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606390, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:51.042821 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 23496
I20260928 14:39:51.042971 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:39:51.043308 6239 external_mini_cluster.cc:1428] Setting key bd0a61fe3b9afc083f7fd8ee74cd90cf
I20260928 14:39:51.046253 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.034973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:57841 (local address 127.6.23.254:34495)
0928 14:39:51.035134 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.035139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.036048 (+ 909us) server_negotiation.cc:407] Connection header received
0928 14:39:51.036905 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.036907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.036951 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.037025 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:51.038162 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.038669 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.039821 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.040043 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.040119 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.042485 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:51.042488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:51.042491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:51.042513 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:51.044190 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:51.044718 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:51.044723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:51.044727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:51.044823 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:51.045196 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:51.045200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:51.045201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:51.045390 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:51.045488 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.045977 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.046099 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:39:51.047129 23627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.035456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34495 (local address 127.6.23.194:57841)
0928 14:39:51.035896 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.035939 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:51.036708 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:51.037179 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:51.037185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:51.037564 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:51.038043 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.038056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.038769 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.038771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.039723 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.039733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.040166 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.040168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.040327 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.040872 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:51.040887 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:51.042328 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:51.044298 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:51.044305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:51.044321 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:51.044635 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:51.044932 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:51.044935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:51.044937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:51.045094 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:51.045488 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:51.045496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:51.045742 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 14:39:51.045925 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 14:39:51.046257 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
I20260928 14:39:51.048341 23625 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34495
I20260928 14:39:51.048610 23625 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:51.049269 23625 heartbeater.cc:511] Master 127.6.23.254:34495 requested a full tablet report, sending...
I20260928 14:39:51.050468 23300 ts_manager.cc:194] Registered new tserver with Master: 8b9a8a51b8be44708eda946b3821b733 (127.6.23.194:44227)
I20260928 14:39:51.051036 23300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:57841
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:39:51.070041 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34495
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:51.183964 23630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:51.184248 23630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:51.184291 23630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:51.188462 23630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:51.188526 23630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:51.188587 23630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:39:51.193928 23630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:51.194975 23630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:51.196100 23630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:51.205941 23635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:51.205899 23636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:51.206054 23630 server_base.cc:1034] running on GCE node
W20260928 14:39:51.205893 23638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:51.206612 23630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:51.207357 23630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:51.208551 23630 hybrid_clock.cc:648] HybridClock initialized: now 1790606391208532 us; error 32 us; skew 500 ppm
Sep 28 14:39:51 dist-test-slave-06lh krb5kdc[23255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606391, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:51.211759 23630 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:39:51.212773 23630 webserver.cc:460] Webserver started at http://127.6.23.195:33191/ using document root <none> and password file <none>
I20260928 14:39:51.213378 23630 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:51.213470 23630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:51.213703 23630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:51.215976 23630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "ce633270ff764209990ede402434a3c8"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "ae2a960f9d6a51588c3a969430f3ae50"
server_key_iv: "44263e7eb9ce7c220d1d98f29a960594"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.216572 23630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "ce633270ff764209990ede402434a3c8"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "ae2a960f9d6a51588c3a969430f3ae50"
server_key_iv: "44263e7eb9ce7c220d1d98f29a960594"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.220613 23630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:39:51.223389 23645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.224711 23630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:39:51.224876 23630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "ce633270ff764209990ede402434a3c8"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "ae2a960f9d6a51588c3a969430f3ae50"
server_key_iv: "44263e7eb9ce7c220d1d98f29a960594"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.225003 23630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:51.234921 23630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:51.238018 23630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:51.238224 23630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:51.238834 23630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:51.239743 23630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:51.239817 23630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.239887 23630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:51.239938 23630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.249351 23630 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:37287
I20260928 14:39:51.249368 23758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:37287 every 8 connection(s)
I20260928 14:39:51.250425 23630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:39:51.256819 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 23630
I20260928 14:39:51.256985 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:39:51.257266 6239 external_mini_cluster.cc:1428] Setting key 8400bc25b7407b72a610bcbe1ad9847a
Sep 28 14:39:51 dist-test-slave-06lh krb5kdc[23255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606391, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:51.264293 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.252455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:57379 (local address 127.6.23.254:34495)
0928 14:39:51.252665 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.252668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.253746 (+ 1078us) server_negotiation.cc:407] Connection header received
0928 14:39:51.254674 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.254678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.254738 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.254820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:51.256100 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.256653 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.258039 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.258372 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.258485 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.261199 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:51.261202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:51.261204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:51.261230 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:51.262446 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:51.262949 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:51.262951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:51.262952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:51.263022 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:51.263356 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:51.263359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:51.263360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:51.263531 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:51.263606 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.264074 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.264173 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260928 14:39:51.265158 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.252861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34495 (local address 127.6.23.195:57379)
0928 14:39:51.253566 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.253610 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:51.254460 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:51.254998 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:51.255005 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:51.255442 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:51.255950 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.255964 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.256793 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.256796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.257916 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.257926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.258536 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.258541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.258751 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.259485 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:51.259509 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:51.261043 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:51.262546 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:51.262551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:51.262564 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:51.262828 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:51.263094 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:51.263096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:51.263097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:51.263275 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:51.263611 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:51.263616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:51.263869 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 14:39:51.264020 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 14:39:51.264282 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":119,"threads_started":1}
I20260928 14:39:51.266541 23759 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34495
I20260928 14:39:51.266795 23759 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:51.267454 23759 heartbeater.cc:511] Master 127.6.23.254:34495 requested a full tablet report, sending...
I20260928 14:39:51.268518 23300 ts_manager.cc:194] Registered new tserver with Master: ce633270ff764209990ede402434a3c8 (127.6.23.195:37287)
I20260928 14:39:51.269083 23300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:57379
I20260928 14:39:51.271483 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:39:51.280894 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.273487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34458 (local address 127.6.23.254:34495)
0928 14:39:51.273607 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.273611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.273706 (+ 95us) server_negotiation.cc:407] Connection header received
0928 14:39:51.273809 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.273811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.273854 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.273920 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:51.274610 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.275130 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.276039 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.276259 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.276343 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.277793 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:51.277809 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:51.277815 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:51.277845 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:51.279484 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:51.279872 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:51.279875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:51.279876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:51.279948 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:51.280166 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:51.280168 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:51.280169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:51.280377 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:51.280479 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.280680 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.280776 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:39:51.283815 23300 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:39:51.286957 23300 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:39:51.302120 23772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.295511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37287 (local address 127.0.0.1:44940)
0928 14:39:51.296611 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.296625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:51.296724 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:51.297223 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:51.297228 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:51.297253 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:51.297403 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.297409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.299197 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.299200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.300422 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.300428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.301524 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.301528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.301639 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.301680 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:39:51.301803 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:39:51.301933 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":44,"thread_start_us":110,"threads_started":1}
I20260928 14:39:51.302122 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.295909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.6.23.195:37287)
0928 14:39:51.296312 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.296325 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.296728 (+ 403us) server_negotiation.cc:407] Connection header received
0928 14:39:51.296847 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.296854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.297060 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.297215 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:51.297961 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.299044 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.300532 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.301390 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.301500 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.301648 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.301845 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.301970 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":92,"threads_started":1}
I20260928 14:39:51.302270 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.295514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.6.23.194:44227)
0928 14:39:51.295995 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.296010 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.296419 (+ 409us) server_negotiation.cc:407] Connection header received
0928 14:39:51.296725 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.296731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.296970 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.297113 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:51.297514 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.298765 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.300532 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.301390 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.301500 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.301648 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.302002 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.302096 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":114,"threads_started":1}
I20260928 14:39:51.302498 23774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.296061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41753 (local address 127.0.0.1:34140)
0928 14:39:51.296914 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.296926 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:51.297017 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:51.297718 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:51.297721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:51.297740 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:51.297832 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.297837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.298805 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.298810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.301068 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.301075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.302077 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.302080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.302177 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.302204 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:39:51.302255 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:39:51.302340 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":1174,"thread_start_us":75,"threads_started":1}
I20260928 14:39:51.302667 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.295372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:44227 (local address 127.0.0.1:38304)
0928 14:39:51.296275 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.296329 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:51.296534 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:51.297321 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:51.297324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:51.297336 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:51.297405 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.297409 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.298872 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.298874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.300400 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.300412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.301730 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.301733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.301844 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.301868 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:39:51.302442 (+ 574us) client_negotiation.cc:240] Negotiation successful
0928 14:39:51.302524 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":56,"thread_start_us":66,"threads_started":1}
I20260928 14:39:51.303104 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.296462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.6.23.193:41753)
0928 14:39:51.297281 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.297297 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.297320 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:39:51.297397 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.297401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.297582 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.297725 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:51.297939 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.298670 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.301179 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.301948 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.302628 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.302786 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.302845 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.302941 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":88,"threads_started":1}
I20260928 14:39:51.305689 23559 tablet_service.cc:1467] Processing CreateTablet for tablet 5a15211963f74a1e9a772ec562044899 (DEFAULT_TABLE table=test-table [id=563d603a2da54fc1b11b6166d9c55c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:51.305864 23693 tablet_service.cc:1467] Processing CreateTablet for tablet 5a15211963f74a1e9a772ec562044899 (DEFAULT_TABLE table=test-table [id=563d603a2da54fc1b11b6166d9c55c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:51.305864 23425 tablet_service.cc:1467] Processing CreateTablet for tablet 5a15211963f74a1e9a772ec562044899 (DEFAULT_TABLE table=test-table [id=563d603a2da54fc1b11b6166d9c55c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:51.306704 23559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a15211963f74a1e9a772ec562044899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:51.306879 23693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a15211963f74a1e9a772ec562044899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:51.306875 23425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a15211963f74a1e9a772ec562044899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:51.313544 23776 tablet_bootstrap.cc:492] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: Bootstrap starting.
I20260928 14:39:51.314699 23777 tablet_bootstrap.cc:492] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: Bootstrap starting.
I20260928 14:39:51.314795 23778 tablet_bootstrap.cc:492] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384: Bootstrap starting.
I20260928 14:39:51.316354 23776 tablet_bootstrap.cc:654] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:51.316947 23778 tablet_bootstrap.cc:654] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:51.316947 23777 tablet_bootstrap.cc:654] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:51.317169 23776 log.cc:824] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:51.318233 23778 log.cc:824] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:51.318233 23777 log.cc:824] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:51.319696 23776 tablet_bootstrap.cc:492] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: No bootstrap required, opened a new log
I20260928 14:39:51.319917 23776 ts_tablet_manager.cc:1397] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:39:51.320050 23778 tablet_bootstrap.cc:492] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384: No bootstrap required, opened a new log
I20260928 14:39:51.320211 23777 tablet_bootstrap.cc:492] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: No bootstrap required, opened a new log
I20260928 14:39:51.320251 23778 ts_tablet_manager.cc:1397] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:39:51.320389 23777 ts_tablet_manager.cc:1397] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:39:51.323210 23778 raft_consensus.cc:348] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.323544 23778 raft_consensus.cc:374] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:51.323622 23778 raft_consensus.cc:729] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0ee8009c13443fb2ab33e93a164384, State: Initialized, Role: FOLLOWER
I20260928 14:39:51.324093 23778 consensus_queue.cc:260] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.324054 23777 raft_consensus.cc:348] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.324312 23777 raft_consensus.cc:374] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:51.324373 23777 raft_consensus.cc:729] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce633270ff764209990ede402434a3c8, State: Initialized, Role: FOLLOWER
I20260928 14:39:51.324774 23491 heartbeater.cc:503] Master 127.6.23.254:34495 was elected leader, sending a full tablet report...
I20260928 14:39:51.324903 23777 consensus_queue.cc:260] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.325156 23778 ts_tablet_manager.cc:1428] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260928 14:39:51.323997 23776 raft_consensus.cc:348] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.325409 23776 raft_consensus.cc:374] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:51.325501 23776 raft_consensus.cc:729] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b9a8a51b8be44708eda946b3821b733, State: Initialized, Role: FOLLOWER
I20260928 14:39:51.325799 23759 heartbeater.cc:503] Master 127.6.23.254:34495 was elected leader, sending a full tablet report...
I20260928 14:39:51.325984 23776 consensus_queue.cc:260] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.326635 23625 heartbeater.cc:503] Master 127.6.23.254:34495 was elected leader, sending a full tablet report...
I20260928 14:39:51.326964 23776 ts_tablet_manager.cc:1428] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:39:51.327585 23777 ts_tablet_manager.cc:1428] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:39:51.351445 23782 raft_consensus.cc:482] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:51.351693 23782 raft_consensus.cc:504] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.352857 23782 leader_election.cc:288] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b9a8a51b8be44708eda946b3821b733 (127.6.23.194:44227), ce633270ff764209990ede402434a3c8 (127.6.23.195:37287)
I20260928 14:39:51.357281 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.353287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:36797 (local address 127.6.23.194:44227)
0928 14:39:51.353434 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.353439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.353551 (+ 112us) server_negotiation.cc:407] Connection header received
0928 14:39:51.353793 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.353798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.353854 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.353923 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:51.354241 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.354767 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.356117 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.356634 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.356770 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.356828 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.357096 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.357169 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:39:51.357281 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.353274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:36083 (local address 127.6.23.195:37287)
0928 14:39:51.353506 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.353513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.353681 (+ 168us) server_negotiation.cc:407] Connection header received
0928 14:39:51.353827 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.353829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.353871 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.353961 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:51.354276 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.354771 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.356055 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.356607 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.356741 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.356796 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.357059 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.357149 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260928 14:39:51.357264 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.353158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:44227 (local address 127.6.23.193:36797)
0928 14:39:51.353457 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.353472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:51.353638 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:51.353965 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:51.353968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:51.353984 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:51.354130 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.354135 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.354884 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.354891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.356001 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.356014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.356869 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.356872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.356983 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.357003 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:39:51.357067 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:39:51.357114 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":84,"spinlock_wait_cycles":24832,"thread_start_us":74,"threads_started":1}
I20260928 14:39:51.357264 23786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.353158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37287 (local address 127.6.23.193:36083)
0928 14:39:51.353584 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.353595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:51.353690 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:51.353987 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:51.353990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:51.354036 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:51.354150 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.354154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.354907 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.354910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.355923 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.355933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.356734 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.356738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.356850 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.356875 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:39:51.356986 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:39:51.357080 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":67,"spinlock_wait_cycles":3200,"thread_start_us":133,"threads_started":1}
I20260928 14:39:51.357934 23713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a15211963f74a1e9a772ec562044899" candidate_uuid: "8d0ee8009c13443fb2ab33e93a164384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce633270ff764209990ede402434a3c8" is_pre_election: true
I20260928 14:39:51.357932 23579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a15211963f74a1e9a772ec562044899" candidate_uuid: "8d0ee8009c13443fb2ab33e93a164384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9a8a51b8be44708eda946b3821b733" is_pre_election: true
I20260928 14:39:51.358273 23713 raft_consensus.cc:2454] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d0ee8009c13443fb2ab33e93a164384 in term 0.
I20260928 14:39:51.358273 23579 raft_consensus.cc:2454] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d0ee8009c13443fb2ab33e93a164384 in term 0.
I20260928 14:39:51.358773 23379 leader_election.cc:302] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b9a8a51b8be44708eda946b3821b733, 8d0ee8009c13443fb2ab33e93a164384; no voters:
I20260928 14:39:51.359041 23782 raft_consensus.cc:2781] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:39:51.359135 23782 raft_consensus.cc:482] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:39:51.359181 23782 raft_consensus.cc:3037] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:51.360288 23782 raft_consensus.cc:504] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.360662 23782 leader_election.cc:288] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [CANDIDATE]: Term 1 election: Requested vote from peers 8b9a8a51b8be44708eda946b3821b733 (127.6.23.194:44227), ce633270ff764209990ede402434a3c8 (127.6.23.195:37287)
I20260928 14:39:51.361028 23579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a15211963f74a1e9a772ec562044899" candidate_uuid: "8d0ee8009c13443fb2ab33e93a164384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b9a8a51b8be44708eda946b3821b733"
I20260928 14:39:51.361079 23713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a15211963f74a1e9a772ec562044899" candidate_uuid: "8d0ee8009c13443fb2ab33e93a164384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce633270ff764209990ede402434a3c8"
I20260928 14:39:51.361148 23579 raft_consensus.cc:3037] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:51.361227 23713 raft_consensus.cc:3037] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:51.362293 23713 raft_consensus.cc:2454] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d0ee8009c13443fb2ab33e93a164384 in term 1.
I20260928 14:39:51.362298 23579 raft_consensus.cc:2454] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d0ee8009c13443fb2ab33e93a164384 in term 1.
I20260928 14:39:51.362633 23380 leader_election.cc:302] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d0ee8009c13443fb2ab33e93a164384, ce633270ff764209990ede402434a3c8; no voters:
I20260928 14:39:51.362835 23782 raft_consensus.cc:2781] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:51.363080 23782 raft_consensus.cc:686] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [term 1 LEADER]: Becoming Leader. State: Replica: 8d0ee8009c13443fb2ab33e93a164384, State: Running, Role: LEADER
I20260928 14:39:51.363488 23782 consensus_queue.cc:237] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } }
I20260928 14:39:51.366729 23300 catalog_manager.cc:5495] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d0ee8009c13443fb2ab33e93a164384 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "8d0ee8009c13443fb2ab33e93a164384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d0ee8009c13443fb2ab33e93a164384" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41753 } health_report { overall_health: HEALTHY } } }
I20260928 14:39:51.380558 23775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.377169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.6.23.193:41753)
0928 14:39:51.377293 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.377297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.377327 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:39:51.377402 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.377405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.377460 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.377544 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:39:51.377761 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.378264 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.379083 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.379391 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.379479 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.379752 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:39:51.380187 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:39:51.380283 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.380334 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.380385 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:39:51.386490 23713 raft_consensus.cc:1261] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 1 FOLLOWER]: Refusing update from remote peer 8d0ee8009c13443fb2ab33e93a164384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:51.386490 23579 raft_consensus.cc:1261] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 1 FOLLOWER]: Refusing update from remote peer 8d0ee8009c13443fb2ab33e93a164384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:51.387279 23787 consensus_queue.cc:1035] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b9a8a51b8be44708eda946b3821b733" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:51.387509 23782 consensus_queue.cc:1035] T 5a15211963f74a1e9a772ec562044899 P 8d0ee8009c13443fb2ab33e93a164384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce633270ff764209990ede402434a3c8" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:51.395871 23791 mvcc.cc:205] Tried to move back new op lower bound from 7334323779112022016 to 7334323779027693568. Current Snapshot: MvccSnapshot[applied={T|T < 7334323779112022016}]
I20260928 14:39:51.396556 23792 mvcc.cc:205] Tried to move back new op lower bound from 7334323779112022016 to 7334323779027693568. Current Snapshot: MvccSnapshot[applied={T|T < 7334323779112022016}]
I20260928 14:39:51.404439 23299 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:51.404603 23299 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:51.407078 23299 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f71d559556d146c18bb7fb0b82110d87: Sending DeleteTablet for 3 replicas of tablet 5a15211963f74a1e9a772ec562044899
I20260928 14:39:51.407941 23693 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a15211963f74a1e9a772ec562044899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:51 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:44940
I20260928 14:39:51.407995 23559 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a15211963f74a1e9a772ec562044899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:51 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:38304
I20260928 14:39:51.408120 23425 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a15211963f74a1e9a772ec562044899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:51 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:34140
I20260928 14:39:51.408564 23800 tablet_replica.cc:330] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: stopping tablet replica
I20260928 14:39:51.408955 23800 raft_consensus.cc:2229] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:51.409274 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 23362
I20260928 14:39:51.409296 23800 raft_consensus.cc:2258] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:51.410364 23801 tablet_replica.cc:330] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: stopping tablet replica
I20260928 14:39:51.410770 23801 raft_consensus.cc:2229] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:51.411082 23801 raft_consensus.cc:2258] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:51.411391 23800 ts_tablet_manager.cc:1905] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:51.413362 23801 ts_tablet_manager.cc:1905] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:51.415010 23800 ts_tablet_manager.cc:1918] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:51.415124 23800 log.cc:1196] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/5a15211963f74a1e9a772ec562044899
I20260928 14:39:51.415556 23800 ts_tablet_manager.cc:1939] T 5a15211963f74a1e9a772ec562044899 P ce633270ff764209990ede402434a3c8: Deleting consensus metadata
I20260928 14:39:51.415764 23801 ts_tablet_manager.cc:1918] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:51.415870 23801 log.cc:1196] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/5a15211963f74a1e9a772ec562044899
I20260928 14:39:51.416177 23801 ts_tablet_manager.cc:1939] T 5a15211963f74a1e9a772ec562044899 P 8b9a8a51b8be44708eda946b3821b733: Deleting consensus metadata
I20260928 14:39:51.417001 23286 catalog_manager.cc:4841] TS ce633270ff764209990ede402434a3c8 (127.6.23.195:37287): tablet 5a15211963f74a1e9a772ec562044899 (table test-table [id=563d603a2da54fc1b11b6166d9c55c91]) successfully deleted
I20260928 14:39:51.417193 23285 catalog_manager.cc:4841] TS 8b9a8a51b8be44708eda946b3821b733 (127.6.23.194:44227): tablet 5a15211963f74a1e9a772ec562044899 (table test-table [id=563d603a2da54fc1b11b6166d9c55c91]) successfully deleted
W20260928 14:39:51.418255 23285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:41753 (error 108)
I20260928 14:39:51.418522 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 23496
I20260928 14:39:51.424088 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.423816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41753 (local address 127.0.0.1:34166)
0928 14:39:51.423942 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.424008 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:41753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 14:39:51.424393 23285 catalog_manager.cc:4584] TS 8d0ee8009c13443fb2ab33e93a164384 (127.6.23.193:41753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a15211963f74a1e9a772ec562044899: Network error: Client connection negotiation failed: client connection to 127.6.23.193:41753: connect: Connection refused (error 111)
I20260928 14:39:51.424988 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 23630
I20260928 14:39:51.431126 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 23268
2026-09-28T14:39:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2247 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T14:39:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:39:51Z Disabled control of system clock
I20260928 14:39:51.462850 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40145
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:40145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:39:51.574563 23807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:51.574853 23807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:51.574896 23807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:51.578933 23807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:39:51.578991 23807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:51.579007 23807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:51.579025 23807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:39:51.579041 23807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:39:51.583770 23807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:40145
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:51.584823 23807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:51.586220 23807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:51.594817 23812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:51.594832 23813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:51.594838 23815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:51.595166 23807 server_base.cc:1034] running on GCE node
I20260928 14:39:51.595788 23807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:51.596894 23807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:51.598057 23807 hybrid_clock.cc:648] HybridClock initialized: now 1790606391598038 us; error 31 us; skew 500 ppm
I20260928 14:39:51.599908 23807 webserver.cc:460] Webserver started at http://127.6.23.254:39659/ using document root <none> and password file <none>
I20260928 14:39:51.600441 23807 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:51.600502 23807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:51.600680 23807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:51.602749 23807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "b0430942d98d4a489a5026ab27f9e8de"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "e6c271b9ea2b578f2052a1fc2f195d1e"
server_key_iv: "2f3a8a1f89668144767802eccbf9daf2"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.603353 23807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "b0430942d98d4a489a5026ab27f9e8de"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "e6c271b9ea2b578f2052a1fc2f195d1e"
server_key_iv: "2f3a8a1f89668144767802eccbf9daf2"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.607532 23807 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:39:51.610033 23821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.611405 23807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:39:51.611542 23807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "b0430942d98d4a489a5026ab27f9e8de"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "e6c271b9ea2b578f2052a1fc2f195d1e"
server_key_iv: "2f3a8a1f89668144767802eccbf9daf2"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.611668 23807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:51.622330 23807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:51.623090 23807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:51.623360 23807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:51.630498 23807 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:40145
I20260928 14:39:51.630517 23873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:40145 every 8 connection(s)
I20260928 14:39:51.631609 23807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:39:51.635104 23874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:51.638688 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 23807
I20260928 14:39:51.638787 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:39:51.639094 6239 external_mini_cluster.cc:1428] Setting key cce85b93c0017da50a788bd605337734
I20260928 14:39:51.641034 23874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de: Bootstrap starting.
I20260928 14:39:51.643486 23874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:51.644330 23874 log.cc:824] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:51.646597 23874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de: No bootstrap required, opened a new log
I20260928 14:39:51.648447 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.640482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.6.23.254:40145)
0928 14:39:51.641112 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.641129 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.641189 (+ 60us) server_negotiation.cc:407] Connection header received
0928 14:39:51.641847 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.641863 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.642255 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.642546 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:51.642888 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.643972 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.645203 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.645666 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.645780 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.646638 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:51.646643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:51.646678 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:51.647089 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:51.647156 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:51.647177 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:51.647313 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.647441 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.647722 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
I20260928 14:39:51.650347 23874 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0430942d98d4a489a5026ab27f9e8de" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40145 } }
I20260928 14:39:51.650561 23874 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:51.650617 23874 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0430942d98d4a489a5026ab27f9e8de, State: Initialized, Role: FOLLOWER
I20260928 14:39:51.651021 23874 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0430942d98d4a489a5026ab27f9e8de" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40145 } }
I20260928 14:39:51.651156 23874 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:39:51.651268 23874 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:39:51.651388 23874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:51.652331 23874 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0430942d98d4a489a5026ab27f9e8de" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40145 } }
I20260928 14:39:51.652669 23874 leader_election.cc:302] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0430942d98d4a489a5026ab27f9e8de; no voters:
I20260928 14:39:51.652984 23874 leader_election.cc:288] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:39:51.653095 23879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:51.653309 23879 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [term 1 LEADER]: Becoming Leader. State: Replica: b0430942d98d4a489a5026ab27f9e8de, State: Running, Role: LEADER
I20260928 14:39:51.653635 23879 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0430942d98d4a489a5026ab27f9e8de" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40145 } }
I20260928 14:39:51.654225 23874 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:39:51.655330 23881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0430942d98d4a489a5026ab27f9e8de. Latest consensus state: current_term: 1 leader_uuid: "b0430942d98d4a489a5026ab27f9e8de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0430942d98d4a489a5026ab27f9e8de" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40145 } } }
I20260928 14:39:51.655465 23881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:51.655736 23880 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0430942d98d4a489a5026ab27f9e8de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0430942d98d4a489a5026ab27f9e8de" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40145 } } }
I20260928 14:39:51.655831 23880 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:51.655849 23888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:39:51.658725 23888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:39:51.663899 23888 catalog_manager.cc:1347] Generated new cluster ID: 381addc77c7b452a93bf7a2733492cb1
I20260928 14:39:51.663971 23888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:39:51.672820 23888 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:39:51.674093 23888 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:39:51.683175 23888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de: Generated new TSK 0
I20260928 14:39:51.684070 23888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:39:51.697535 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40145
--builtin_ntp_servers=127.6.23.212:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:39:51.812188 23898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:51.812474 23898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:51.812513 23898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:51.816754 23898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:51.816820 23898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:51.816907 23898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:39:51.821869 23898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:51.823104 23898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:51.824584 23898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:51.834441 23904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:51.834441 23906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:51.834568 23898 server_base.cc:1034] running on GCE node
W20260928 14:39:51.834439 23903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:51.835207 23898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:51.835875 23898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:51.837036 23898 hybrid_clock.cc:648] HybridClock initialized: now 1790606391837019 us; error 30 us; skew 500 ppm
I20260928 14:39:51.838884 23898 webserver.cc:460] Webserver started at http://127.6.23.193:38679/ using document root <none> and password file <none>
I20260928 14:39:51.839524 23898 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:51.839587 23898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:51.839819 23898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:51.841868 23898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "b32c2c5444af440493b1ea056447dcb3"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "d08509df7586ea06c34302eaa4e9000d"
server_key_iv: "e663d469325a70ec95ce373bdd5bfc80"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.842427 23898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "b32c2c5444af440493b1ea056447dcb3"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "d08509df7586ea06c34302eaa4e9000d"
server_key_iv: "e663d469325a70ec95ce373bdd5bfc80"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.846179 23898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20260928 14:39:51.848744 23912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.850036 23898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:39:51.850168 23898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "b32c2c5444af440493b1ea056447dcb3"
format_stamp: "Formatted at 2026-09-28 14:39:51 on dist-test-slave-06lh"
server_key: "d08509df7586ea06c34302eaa4e9000d"
server_key_iv: "e663d469325a70ec95ce373bdd5bfc80"
server_key_version: "encryptionkey@0"
I20260928 14:39:51.850286 23898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:51.863495 23898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:51.864192 23898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:51.864380 23898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:51.864966 23898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:51.865906 23898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:51.865973 23898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.866044 23898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:51.866082 23898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:51.874927 23898 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:42479
I20260928 14:39:51.874944 24025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:42479 every 8 connection(s)
I20260928 14:39:51.875968 23898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:39:51.884117 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 23898
I20260928 14:39:51.884325 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:39:51.884734 6239 external_mini_cluster.cc:1428] Setting key faaf23f55facc02ce96928c08ec32a27
I20260928 14:39:51.885550 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.877889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:44945 (local address 127.6.23.254:40145)
0928 14:39:51.878039 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:51.878043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:51.879153 (+ 1110us) server_negotiation.cc:407] Connection header received
0928 14:39:51.880631 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:51.880634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:51.880696 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:51.880801 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:51.881720 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.882278 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.883491 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.883731 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.883866 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.885049 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:51.885053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:51.885055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:51.885073 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:51.885092 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:51.885100 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:51.885170 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:51.885339 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:39:51.885403 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:39:51.886417 24028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:51.878311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40145 (local address 127.6.23.193:44945)
0928 14:39:51.878987 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:51.879030 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:51.880408 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:51.880991 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:51.880997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:51.881086 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:51.881572 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.881585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.882416 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.882420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.883358 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:51.883367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:51.883891 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:51.883896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:51.884074 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:51.884739 (+ 665us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:51.884741 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:51.884789 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:51.884879 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:51.885184 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:51.885195 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:51.885200 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:39:51.885328 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 14:39:51.885617 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260928 14:39:51.886919 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40145
--builtin_ntp_servers=127.6.23.212:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:39:51.887660 24026 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40145
I20260928 14:39:51.887920 24026 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:51.888554 24026 heartbeater.cc:511] Master 127.6.23.254:40145 requested a full tablet report, sending...
I20260928 14:39:51.890125 23838 ts_manager.cc:194] Registered new tserver with Master: b32c2c5444af440493b1ea056447dcb3 (127.6.23.193:42479)
I20260928 14:39:51.891353 23838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.193:44945
W20260928 14:39:52.001664 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:52.001978 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:52.002046 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:52.006299 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:52.006390 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:52.006475 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:39:52.011444 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:52.012606 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:52.014081 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:52.024187 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:52.024168 24029 server_base.cc:1034] running on GCE node
W20260928 14:39:52.024106 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:52.024125 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:52.024820 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:52.025508 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:52.026672 24029 hybrid_clock.cc:648] HybridClock initialized: now 1790606392026647 us; error 35 us; skew 500 ppm
I20260928 14:39:52.028586 24029 webserver.cc:460] Webserver started at http://127.6.23.194:46245/ using document root <none> and password file <none>
I20260928 14:39:52.029193 24029 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:52.029263 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:52.029500 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:52.031795 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "ab70114038124189acba56782ff307a7"
format_stamp: "Formatted at 2026-09-28 14:39:52 on dist-test-slave-06lh"
server_key: "96ac2bfe8f43f0c4a19bbffbef159351"
server_key_iv: "0d5d5e15ccc0dee3904c5ae281c64c24"
server_key_version: "encryptionkey@0"
I20260928 14:39:52.032354 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "ab70114038124189acba56782ff307a7"
format_stamp: "Formatted at 2026-09-28 14:39:52 on dist-test-slave-06lh"
server_key: "96ac2bfe8f43f0c4a19bbffbef159351"
server_key_iv: "0d5d5e15ccc0dee3904c5ae281c64c24"
server_key_version: "encryptionkey@0"
I20260928 14:39:52.036092 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:39:52.038554 24043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:52.039932 24029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 14:39:52.040078 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "ab70114038124189acba56782ff307a7"
format_stamp: "Formatted at 2026-09-28 14:39:52 on dist-test-slave-06lh"
server_key: "96ac2bfe8f43f0c4a19bbffbef159351"
server_key_iv: "0d5d5e15ccc0dee3904c5ae281c64c24"
server_key_version: "encryptionkey@0"
I20260928 14:39:52.040202 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:52.055612 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:52.056393 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:52.056596 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:52.057207 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:52.058179 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:52.058247 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:52.058314 24029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:52.058363 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:52.067292 24029 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:37577
I20260928 14:39:52.067323 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:37577 every 8 connection(s)
I20260928 14:39:52.068229 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:39:52.074059 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 24029
I20260928 14:39:52.074167 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:39:52.074455 6239 external_mini_cluster.cc:1428] Setting key bc8601d4a569daee8bb195d1c53fb97b
I20260928 14:39:52.076650 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40145
--builtin_ntp_servers=127.6.23.212:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:39:52.078100 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.070166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:37751 (local address 127.6.23.254:40145)
0928 14:39:52.070345 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.070350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.071469 (+ 1119us) server_negotiation.cc:407] Connection header received
0928 14:39:52.072819 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.072822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.072876 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.072951 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:52.073879 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.074543 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.075990 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.076218 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.076299 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.077599 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:52.077602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:52.077604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:52.077616 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:52.077633 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:52.077641 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:52.077699 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.077884 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.077935 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:39:52.078984 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.070585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40145 (local address 127.6.23.194:37751)
0928 14:39:52.071322 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:52.071364 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:52.072631 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:52.073110 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:52.073117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:52.073199 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:52.073709 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.073722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.074670 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.074675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.075877 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.075886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.076380 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.076386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.076603 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.077287 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:52.077291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:52.077352 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:52.077440 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:52.077722 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:52.077733 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:52.077742 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:39:52.077880 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 14:39:52.078214 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":112,"threads_started":1}
I20260928 14:39:52.080123 24157 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40145
I20260928 14:39:52.080376 24157 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:52.081038 24157 heartbeater.cc:511] Master 127.6.23.254:40145 requested a full tablet report, sending...
I20260928 14:39:52.082202 23838 ts_manager.cc:194] Registered new tserver with Master: ab70114038124189acba56782ff307a7 (127.6.23.194:37577)
I20260928 14:39:52.082787 23838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.194:37751
W20260928 14:39:52.192719 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:52.192993 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:52.193033 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:52.197166 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:52.197232 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:52.197314 24160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:39:52.202193 24160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:40145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:52.203399 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:52.204855 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:52.214871 24166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:52.214895 24168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:52.215018 24160 server_base.cc:1034] running on GCE node
W20260928 14:39:52.214885 24165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:52.215626 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:52.216406 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:52.217608 24160 hybrid_clock.cc:648] HybridClock initialized: now 1790606392217587 us; error 31 us; skew 500 ppm
I20260928 14:39:52.219753 24160 webserver.cc:460] Webserver started at http://127.6.23.195:32943/ using document root <none> and password file <none>
I20260928 14:39:52.220315 24160 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:52.220379 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:52.220602 24160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:52.222756 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c"
format_stamp: "Formatted at 2026-09-28 14:39:52 on dist-test-slave-06lh"
server_key: "8e53bc570136623693269422ae82d434"
server_key_iv: "6d2e54031a241e32a3b435aeb3ca3c07"
server_key_version: "encryptionkey@0"
I20260928 14:39:52.223392 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c"
format_stamp: "Formatted at 2026-09-28 14:39:52 on dist-test-slave-06lh"
server_key: "8e53bc570136623693269422ae82d434"
server_key_iv: "6d2e54031a241e32a3b435aeb3ca3c07"
server_key_version: "encryptionkey@0"
I20260928 14:39:52.227412 24160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:39:52.230016 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:52.231585 24160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:39:52.231707 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c"
format_stamp: "Formatted at 2026-09-28 14:39:52 on dist-test-slave-06lh"
server_key: "8e53bc570136623693269422ae82d434"
server_key_iv: "6d2e54031a241e32a3b435aeb3ca3c07"
server_key_version: "encryptionkey@0"
I20260928 14:39:52.231832 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:52.242125 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:52.242805 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:52.242997 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:52.243652 24160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:52.244656 24160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:52.244710 24160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:52.244752 24160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:52.244774 24160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:52.253623 24160 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:43677
I20260928 14:39:52.253639 24287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:43677 every 8 connection(s)
I20260928 14:39:52.254607 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:39:52.263697 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 24160
I20260928 14:39:52.263840 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:39:52.264178 6239 external_mini_cluster.cc:1428] Setting key a479967d2b1c481cb90cbe0884a8fe1e
I20260928 14:39:52.264254 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.256524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:51297 (local address 127.6.23.254:40145)
0928 14:39:52.256671 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.256675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.257788 (+ 1113us) server_negotiation.cc:407] Connection header received
0928 14:39:52.259215 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.259218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.259305 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.259384 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:52.260291 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.260836 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.262091 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.262352 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.262469 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.263666 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:52.263669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:52.263672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:52.263686 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:52.263714 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:52.263722 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:52.263794 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.264023 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.264097 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:39:52.265173 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.256963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:40145 (local address 127.6.23.195:51297)
0928 14:39:52.257632 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:52.257673 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:52.258995 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:52.259555 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:52.259562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:52.259653 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:52.260140 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.260153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.260965 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.260969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.261872 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.261882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.262514 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.262518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.262702 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.263394 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:52.263395 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:52.263437 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:52.263503 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:52.263808 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:52.263820 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:52.263829 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:39:52.263978 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 14:39:52.264210 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":102,"threads_started":1}
I20260928 14:39:52.266314 24288 heartbeater.cc:347] Connected to a master server at 127.6.23.254:40145
I20260928 14:39:52.266552 24288 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:52.267035 24288 heartbeater.cc:511] Master 127.6.23.254:40145 requested a full tablet report, sending...
I20260928 14:39:52.268123 23838 ts_manager.cc:194] Registered new tserver with Master: 16ff3519234f4dfca2f5ecf3e17b0c2c (127.6.23.195:43677)
I20260928 14:39:52.268659 23838 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.195:51297
I20260928 14:39:52.278235 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:39:52.283715 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.280226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.6.23.254:40145)
0928 14:39:52.280354 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.280357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.280517 (+ 160us) server_negotiation.cc:407] Connection header received
0928 14:39:52.280611 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.280613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.280654 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.280719 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:52.280942 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.281441 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.282270 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.282475 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.282547 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.283368 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:52.283372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:52.283374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:52.283385 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:52.283405 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:52.283413 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:52.283479 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.283555 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.283605 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:39:52.286425 23838 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:39:52.289621 23838 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:39:52.304209 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.298868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.6.23.193:42479)
0928 14:39:52.299322 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.299337 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.299401 (+ 64us) server_negotiation.cc:407] Connection header received
0928 14:39:52.299533 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.299537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.299695 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.299853 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:52.300156 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.301361 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.302842 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.303530 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.303650 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.303864 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.303920 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.304026 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":79,"threads_started":1}
I20260928 14:39:52.304230 24300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.298385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:42479 (local address 127.0.0.1:53628)
0928 14:39:52.299322 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:52.299335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:52.299421 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:52.299848 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:52.299854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:52.299878 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:52.300010 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.300018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.301476 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.301479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.302711 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.302717 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.303649 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.303653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.303759 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.303795 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:39:52.303898 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:39:52.304025 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":133,"spinlock_wait_cycles":4864,"thread_start_us":94,"threads_started":1}
I20260928 14:39:52.304483 24299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.298129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37577 (local address 127.0.0.1:58300)
0928 14:39:52.299315 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:52.299329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:52.299418 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:52.300561 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:52.300564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:52.300582 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:52.300656 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.300661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.301972 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.301975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.302855 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.302865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.304098 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.304101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.304181 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.304201 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:39:52.304280 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:39:52.304359 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"mutex_wait_us":28,"spinlock_wait_cycles":384,"thread_start_us":93,"threads_started":1}
I20260928 14:39:52.304921 24301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.298036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:43677 (local address 127.0.0.1:45752)
0928 14:39:52.298909 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:52.298959 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:52.299080 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:52.300002 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:52.300005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:52.300023 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:52.300125 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.300131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.301216 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.301222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.302706 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.302717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.303854 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.303857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.303943 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.303965 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:39:52.304015 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 14:39:52.304090 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":56,"thread_start_us":94,"threads_started":1}
I20260928 14:39:52.304947 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.298747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.6.23.194:37577)
0928 14:39:52.300134 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.300147 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.300168 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:39:52.300236 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.300240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.300419 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.300561 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:52.300792 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.301853 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.302970 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.303757 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.303856 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.304040 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.304649 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.304749 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1255,"thread_start_us":116,"threads_started":1}
I20260928 14:39:52.305856 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.298327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.6.23.195:43677)
0928 14:39:52.299646 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.299654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.299672 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:39:52.299731 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.299735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.299878 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.300030 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:52.300222 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.301115 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.302863 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.303695 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.305354 (+ 1659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.305549 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.305606 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.305694 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1235,"thread_start_us":171,"threads_started":1}
I20260928 14:39:52.307580 23960 tablet_service.cc:1467] Processing CreateTablet for tablet e39a363dc41447f28f9d6b2f5234a7bf (DEFAULT_TABLE table=test-table [id=e790592bfbf9499dbd3408b61c39c0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:52.307778 24091 tablet_service.cc:1467] Processing CreateTablet for tablet e39a363dc41447f28f9d6b2f5234a7bf (DEFAULT_TABLE table=test-table [id=e790592bfbf9499dbd3408b61c39c0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:52.308713 24091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39a363dc41447f28f9d6b2f5234a7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:52.308706 23960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39a363dc41447f28f9d6b2f5234a7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:52.308619 24222 tablet_service.cc:1467] Processing CreateTablet for tablet e39a363dc41447f28f9d6b2f5234a7bf (DEFAULT_TABLE table=test-table [id=e790592bfbf9499dbd3408b61c39c0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:52.309553 24222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e39a363dc41447f28f9d6b2f5234a7bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:52.314935 24305 tablet_bootstrap.cc:492] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: Bootstrap starting.
I20260928 14:39:52.316556 24307 tablet_bootstrap.cc:492] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Bootstrap starting.
I20260928 14:39:52.317775 24305 tablet_bootstrap.cc:654] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:52.318236 24306 tablet_bootstrap.cc:492] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3: Bootstrap starting.
I20260928 14:39:52.318629 24307 tablet_bootstrap.cc:654] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:52.318660 24305 log.cc:824] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:52.319481 24307 log.cc:824] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:52.320276 24306 tablet_bootstrap.cc:654] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:52.320941 24306 log.cc:824] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:52.321022 24305 tablet_bootstrap.cc:492] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: No bootstrap required, opened a new log
I20260928 14:39:52.321218 24305 ts_tablet_manager.cc:1397] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:39:52.321384 24307 tablet_bootstrap.cc:492] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: No bootstrap required, opened a new log
I20260928 14:39:52.321589 24307 ts_tablet_manager.cc:1397] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:39:52.322892 24306 tablet_bootstrap.cc:492] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3: No bootstrap required, opened a new log
I20260928 14:39:52.323087 24306 ts_tablet_manager.cc:1397] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:39:52.325834 24305 raft_consensus.cc:348] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.325835 24307 raft_consensus.cc:348] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.326131 24307 raft_consensus.cc:374] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:52.326131 24305 raft_consensus.cc:374] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:52.326189 24307 raft_consensus.cc:729] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ff3519234f4dfca2f5ecf3e17b0c2c, State: Initialized, Role: FOLLOWER
I20260928 14:39:52.326200 24305 raft_consensus.cc:729] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab70114038124189acba56782ff307a7, State: Initialized, Role: FOLLOWER
I20260928 14:39:52.326179 24306 raft_consensus.cc:348] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.326443 24306 raft_consensus.cc:374] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:52.326516 24306 raft_consensus.cc:729] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32c2c5444af440493b1ea056447dcb3, State: Initialized, Role: FOLLOWER
I20260928 14:39:52.326686 24307 consensus_queue.cc:260] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.326686 24305 consensus_queue.cc:260] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.326989 24306 consensus_queue.cc:260] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.327621 24288 heartbeater.cc:503] Master 127.6.23.254:40145 was elected leader, sending a full tablet report...
I20260928 14:39:52.327768 24157 heartbeater.cc:503] Master 127.6.23.254:40145 was elected leader, sending a full tablet report...
I20260928 14:39:52.327829 24307 ts_tablet_manager.cc:1428] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:39:52.328011 24306 ts_tablet_manager.cc:1428] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260928 14:39:52.328213 24305 ts_tablet_manager.cc:1428] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260928 14:39:52.328400 24026 heartbeater.cc:503] Master 127.6.23.254:40145 was elected leader, sending a full tablet report...
W20260928 14:39:52.377192 24027 tablet.cc:2369] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:39:52.486795 24312 raft_consensus.cc:482] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:52.487059 24312 raft_consensus.cc:504] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.488400 24312 leader_election.cc:288] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab70114038124189acba56782ff307a7 (127.6.23.194:37577), b32c2c5444af440493b1ea056447dcb3 (127.6.23.193:42479)
I20260928 14:39:52.492435 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.488624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:37577 (local address 127.6.23.195:39591)
0928 14:39:52.488766 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:52.488783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:52.488871 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:52.489313 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:52.489316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:52.489335 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:52.489437 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.489442 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.490314 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.490317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.491303 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.491313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.492029 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.492033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.492132 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.492159 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:39:52.492238 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:39:52.492308 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:39:52.492523 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.488915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:39591 (local address 127.6.23.194:37577)
0928 14:39:52.489042 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.489046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.489061 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:39:52.489112 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.489115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.489167 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.489254 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:52.489548 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.490203 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.491424 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.491920 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.492012 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.492050 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.492295 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.492381 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:39:52.492794 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.488846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:48467 (local address 127.6.23.193:42479)
0928 14:39:52.488982 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.488987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.489535 (+ 548us) server_negotiation.cc:407] Connection header received
0928 14:39:52.489585 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.489587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.489628 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.489683 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:52.489985 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.490619 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.491709 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.492154 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.492284 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.492326 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.492580 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.492660 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:39:52.493137 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.488645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:42479 (local address 127.6.23.195:48467)
0928 14:39:52.489153 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:52.489167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:52.489267 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:52.489726 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:52.489729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:52.489748 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:52.489866 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.489871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.490733 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.490736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.491592 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:52.491602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.492313 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.492317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:52.492412 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.492434 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:39:52.492937 (+ 503us) client_negotiation.cc:240] Negotiation successful
0928 14:39:52.493010 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":129,"thread_start_us":139,"threads_started":1}
I20260928 14:39:52.493166 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39a363dc41447f28f9d6b2f5234a7bf" candidate_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab70114038124189acba56782ff307a7" is_pre_election: true
I20260928 14:39:52.493501 24111 raft_consensus.cc:2454] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ff3519234f4dfca2f5ecf3e17b0c2c in term 0.
I20260928 14:39:52.493857 23980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39a363dc41447f28f9d6b2f5234a7bf" candidate_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32c2c5444af440493b1ea056447dcb3" is_pre_election: true
I20260928 14:39:52.494130 23980 raft_consensus.cc:2454] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ff3519234f4dfca2f5ecf3e17b0c2c in term 0.
I20260928 14:39:52.494127 24177 leader_election.cc:302] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ff3519234f4dfca2f5ecf3e17b0c2c, ab70114038124189acba56782ff307a7; no voters:
I20260928 14:39:52.494441 24312 raft_consensus.cc:2781] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:39:52.494525 24312 raft_consensus.cc:482] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:39:52.494592 24312 raft_consensus.cc:3037] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:52.495676 24312 raft_consensus.cc:504] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.496079 24312 leader_election.cc:288] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [CANDIDATE]: Term 1 election: Requested vote from peers ab70114038124189acba56782ff307a7 (127.6.23.194:37577), b32c2c5444af440493b1ea056447dcb3 (127.6.23.193:42479)
I20260928 14:39:52.496409 24111 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39a363dc41447f28f9d6b2f5234a7bf" candidate_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab70114038124189acba56782ff307a7"
I20260928 14:39:52.496526 23980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e39a363dc41447f28f9d6b2f5234a7bf" candidate_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32c2c5444af440493b1ea056447dcb3"
I20260928 14:39:52.496552 24111 raft_consensus.cc:3037] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:52.496677 23980 raft_consensus.cc:3037] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:52.497635 24111 raft_consensus.cc:2454] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ff3519234f4dfca2f5ecf3e17b0c2c in term 1.
I20260928 14:39:52.497850 23980 raft_consensus.cc:2454] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ff3519234f4dfca2f5ecf3e17b0c2c in term 1.
I20260928 14:39:52.498006 24177 leader_election.cc:302] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ff3519234f4dfca2f5ecf3e17b0c2c, ab70114038124189acba56782ff307a7; no voters:
I20260928 14:39:52.498196 24312 raft_consensus.cc:2781] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:52.498449 24312 raft_consensus.cc:686] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 1 LEADER]: Becoming Leader. State: Replica: 16ff3519234f4dfca2f5ecf3e17b0c2c, State: Running, Role: LEADER
I20260928 14:39:52.498782 24312 consensus_queue.cc:237] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } }
I20260928 14:39:52.502156 23838 catalog_manager.cc:5495] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 16ff3519234f4dfca2f5ecf3e17b0c2c (127.6.23.195). New cstate: current_term: 1 leader_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ff3519234f4dfca2f5ecf3e17b0c2c" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 43677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 } health_report { overall_health: UNKNOWN } } }
W20260928 14:39:52.505554 24289 tablet.cc:2369] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:39:52.542702 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:52.539049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.6.23.195:43677)
0928 14:39:52.539182 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:52.539187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:52.539202 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:39:52.539377 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:52.539379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:52.539437 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:52.539551 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:39:52.539810 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.540342 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.541209 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:52.541434 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:52.541542 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:52.541856 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:39:52.542302 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:39:52.542440 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:52.542483 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 14:39:52.542535 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:39:52.548910 24111 raft_consensus.cc:1261] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 1 FOLLOWER]: Refusing update from remote peer 16ff3519234f4dfca2f5ecf3e17b0c2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:52.548910 23980 raft_consensus.cc:1261] T e39a363dc41447f28f9d6b2f5234a7bf P b32c2c5444af440493b1ea056447dcb3 [term 1 FOLLOWER]: Refusing update from remote peer 16ff3519234f4dfca2f5ecf3e17b0c2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:52.549757 24315 consensus_queue.cc:1035] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab70114038124189acba56782ff307a7" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 37577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:52.549965 24312 consensus_queue.cc:1035] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32c2c5444af440493b1ea056447dcb3" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:52.557224 24318 mvcc.cc:205] Tried to move back new op lower bound from 7334323783871717376 to 7334323783678058496. Current Snapshot: MvccSnapshot[applied={T|T < 7334323783871717376}]
I20260928 14:39:52.559298 24320 mvcc.cc:205] Tried to move back new op lower bound from 7334323783871717376 to 7334323783678058496. Current Snapshot: MvccSnapshot[applied={T|T < 7334323783871717376}]
I20260928 14:39:52.561646 24319 mvcc.cc:205] Tried to move back new op lower bound from 7334323783871717376 to 7334323783678058496. Current Snapshot: MvccSnapshot[applied={T|T < 7334323783871717376 or (T in {7334323783871717376})}]
I20260928 14:39:52.566370 23838 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:52.566546 23838 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:52.569159 23838 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0430942d98d4a489a5026ab27f9e8de: Sending DeleteTablet for 3 replicas of tablet e39a363dc41447f28f9d6b2f5234a7bf
I20260928 14:39:52.570067 24222 tablet_service.cc:1514] Processing DeleteTablet for tablet e39a363dc41447f28f9d6b2f5234a7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:52 UTC) from {username='slave'} at 127.0.0.1:45752
I20260928 14:39:52.570132 24091 tablet_service.cc:1514] Processing DeleteTablet for tablet e39a363dc41447f28f9d6b2f5234a7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:52 UTC) from {username='slave'} at 127.0.0.1:58300
I20260928 14:39:52.570636 24329 tablet_replica.cc:330] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: stopping tablet replica
I20260928 14:39:52.570959 24330 tablet_replica.cc:330] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: stopping tablet replica
I20260928 14:39:52.570976 23960 tablet_service.cc:1514] Processing DeleteTablet for tablet e39a363dc41447f28f9d6b2f5234a7bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:52 UTC) from {username='slave'} at 127.0.0.1:53628
I20260928 14:39:52.571077 24329 raft_consensus.cc:2229] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:39:52.571276 24330 raft_consensus.cc:2229] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:52.571615 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 23898
I20260928 14:39:52.571625 24330 raft_consensus.cc:2258] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:52.571755 24329 raft_consensus.cc:2258] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:52.574129 24329 ts_tablet_manager.cc:1905] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:52.575917 24330 ts_tablet_manager.cc:1905] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:52.577868 24329 ts_tablet_manager.cc:1918] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:52.578050 24329 log.cc:1196] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/e39a363dc41447f28f9d6b2f5234a7bf
I20260928 14:39:52.578452 24329 ts_tablet_manager.cc:1939] T e39a363dc41447f28f9d6b2f5234a7bf P 16ff3519234f4dfca2f5ecf3e17b0c2c: Deleting consensus metadata
I20260928 14:39:52.578513 24330 ts_tablet_manager.cc:1918] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:52.578609 24330 log.cc:1196] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/e39a363dc41447f28f9d6b2f5234a7bf
I20260928 14:39:52.578917 24330 ts_tablet_manager.cc:1939] T e39a363dc41447f28f9d6b2f5234a7bf P ab70114038124189acba56782ff307a7: Deleting consensus metadata
W20260928 14:39:52.579675 23825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:42479 (error 108)
I20260928 14:39:52.580008 23822 catalog_manager.cc:4841] TS 16ff3519234f4dfca2f5ecf3e17b0c2c (127.6.23.195:43677): tablet e39a363dc41447f28f9d6b2f5234a7bf (table test-table [id=e790592bfbf9499dbd3408b61c39c0aa]) successfully deleted
I20260928 14:39:52.580300 23824 catalog_manager.cc:4841] TS ab70114038124189acba56782ff307a7 (127.6.23.194:37577): tablet e39a363dc41447f28f9d6b2f5234a7bf (table test-table [id=e790592bfbf9499dbd3408b61c39c0aa]) successfully deleted
W20260928 14:39:52.580885 24300 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:39:52.580583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:42479 (local address 127.0.0.1:53630)
0928 14:39:52.580713 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:52.580792 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:42479: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 14:39:52.581018 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 24029
W20260928 14:39:52.581110 23825 catalog_manager.cc:4584] TS b32c2c5444af440493b1ea056447dcb3 (127.6.23.193:42479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e39a363dc41447f28f9d6b2f5234a7bf: Network error: Client connection negotiation failed: client connection to 127.6.23.193:42479: connect: Connection reset by peer (error 104)
I20260928 14:39:52.587486 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 24160
I20260928 14:39:52.592772 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 23807
2026-09-28T14:39:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1160 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:39:52 dist-test-slave-06lh krb5kdc[24335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:39:52 dist-test-slave-06lh krb5kdc[24335](info): set up 2 sockets
Sep 28 14:39:52 dist-test-slave-06lh krb5kdc[24335](info): commencing operation
krb5kdc: starting...
W20260928 14:39:53.623394 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:39:53 dist-test-slave-06lh krb5kdc[24335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:39:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:39:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:39:53.744669 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35287
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:35287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:53.859705 24348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:53.860002 24348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:53.860047 24348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:53.864224 24348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:39:53.864296 24348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:53.864316 24348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:53.864333 24348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:39:53.864363 24348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:39:53.869834 24348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:35287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:53.870998 24348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:53.872176 24348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:53.880995 24353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:53.880995 24354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:53.881001 24356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:53.881268 24348 server_base.cc:1034] running on GCE node
I20260928 14:39:53.881986 24348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:53.883064 24348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:53.884220 24348 hybrid_clock.cc:648] HybridClock initialized: now 1790606393884207 us; error 32 us; skew 500 ppm
Sep 28 14:39:53 dist-test-slave-06lh krb5kdc[24335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606393, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:53.886940 24348 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:39:53.887933 24348 webserver.cc:460] Webserver started at http://127.6.23.254:36063/ using document root <none> and password file <none>
I20260928 14:39:53.888513 24348 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:53.888574 24348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:53.888800 24348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:53.890872 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "d4f73378fd7d40ecb8df042f88c71965"
format_stamp: "Formatted at 2026-09-28 14:39:53 on dist-test-slave-06lh"
server_key: "562c20f5352fbe85ca298ca17925abbd"
server_key_iv: "aa8f9848cc36e7ab7444faa1383bfa17"
server_key_version: "encryptionkey@0"
I20260928 14:39:53.891469 24348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "d4f73378fd7d40ecb8df042f88c71965"
format_stamp: "Formatted at 2026-09-28 14:39:53 on dist-test-slave-06lh"
server_key: "562c20f5352fbe85ca298ca17925abbd"
server_key_iv: "aa8f9848cc36e7ab7444faa1383bfa17"
server_key_version: "encryptionkey@0"
I20260928 14:39:53.895344 24348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 14:39:53.897894 24363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:53.899196 24348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:39:53.899405 24348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "d4f73378fd7d40ecb8df042f88c71965"
format_stamp: "Formatted at 2026-09-28 14:39:53 on dist-test-slave-06lh"
server_key: "562c20f5352fbe85ca298ca17925abbd"
server_key_iv: "aa8f9848cc36e7ab7444faa1383bfa17"
server_key_version: "encryptionkey@0"
I20260928 14:39:53.899530 24348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:53.921952 24348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:53.925189 24348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:53.925414 24348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:53.932339 24348 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:35287
I20260928 14:39:53.932365 24415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:35287 every 8 connection(s)
I20260928 14:39:53.933310 24348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:39:53.936021 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:53.941538 24416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965: Bootstrap starting.
I20260928 14:39:53.941773 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 24348
I20260928 14:39:53.941913 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:39:53.942224 6239 external_mini_cluster.cc:1428] Setting key 7c060adf1f0594afe003a68b530f8197
I20260928 14:39:53.944476 24416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:53.945214 24416 log.cc:824] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:53.947103 24416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965: No bootstrap required, opened a new log
I20260928 14:39:53.950744 24416 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f73378fd7d40ecb8df042f88c71965" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35287 } }
I20260928 14:39:53.951046 24416 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:53.951116 24416 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f73378fd7d40ecb8df042f88c71965, State: Initialized, Role: FOLLOWER
Sep 28 14:39:53 dist-test-slave-06lh krb5kdc[24335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:53.951656 24416 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f73378fd7d40ecb8df042f88c71965" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35287 } }
I20260928 14:39:53.951844 24416 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:39:53.951916 24416 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:39:53.952025 24416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:53.953301 24416 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f73378fd7d40ecb8df042f88c71965" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35287 } }
I20260928 14:39:53.953698 24416 leader_election.cc:302] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f73378fd7d40ecb8df042f88c71965; no voters:
I20260928 14:39:53.954073 24416 leader_election.cc:288] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:39:53.954177 24421 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:53.954439 24421 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [term 1 LEADER]: Becoming Leader. State: Replica: d4f73378fd7d40ecb8df042f88c71965, State: Running, Role: LEADER
I20260928 14:39:53.954802 24421 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f73378fd7d40ecb8df042f88c71965" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35287 } }
I20260928 14:39:53.955519 24416 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:39:53.956566 24422 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f73378fd7d40ecb8df042f88c71965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f73378fd7d40ecb8df042f88c71965" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35287 } } }
I20260928 14:39:53.956699 24422 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:53.957137 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f73378fd7d40ecb8df042f88c71965. Latest consensus state: current_term: 1 leader_uuid: "d4f73378fd7d40ecb8df042f88c71965" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f73378fd7d40ecb8df042f88c71965" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35287 } } }
I20260928 14:39:53.957304 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:53.958526 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:53.943669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.6.23.254:35287)
0928 14:39:53.944266 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:53.944280 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:53.944316 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:39:53.945031 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:53.945050 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:53.945442 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:53.945791 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:53.946599 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:53.947857 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:53.948760 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:53.949236 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:53.949400 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:53.952266 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:53.952274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:53.952291 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:53.952328 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:53.954372 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:53.954911 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:53.954917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:53.954924 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:53.955065 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:53.955403 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:53.955406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:53.955408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:53.955918 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:53.956138 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:53.957099 (+ 961us) server_negotiation.cc:299] Negotiation successful
0928 14:39:53.957444 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":151,"threads_started":1}
W20260928 14:39:53.960779 24436 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:39:53.960863 24436 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:39:53.960974 24430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:39:53.961885 24430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:39:53.967672 24430 catalog_manager.cc:1347] Generated new cluster ID: f07a4ccf0bb940958330b07a45eb732e
I20260928 14:39:53.967762 24430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:39:53.974926 24430 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:39:53.976274 24430 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:39:53.981767 24430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965: Generated new TSK 0
I20260928 14:39:53.982590 24430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:39:54.014683 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35287
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:54.129886 24442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:54.130255 24442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:54.130302 24442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:54.134419 24442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:54.134501 24442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:54.134565 24442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:39:54.140053 24442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:54.141207 24442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:54.142225 24442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:54.152171 24447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:54.152141 24450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:54.152323 24442 server_base.cc:1034] running on GCE node
W20260928 14:39:54.152503 24448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:54.152926 24442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:54.153616 24442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:54.154816 24442 hybrid_clock.cc:648] HybridClock initialized: now 1790606394154796 us; error 33 us; skew 500 ppm
Sep 28 14:39:54 dist-test-slave-06lh krb5kdc[24335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606394, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:54.168062 24442 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:39:54.169195 24442 webserver.cc:460] Webserver started at http://127.6.23.193:41703/ using document root <none> and password file <none>
I20260928 14:39:54.169839 24442 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:54.169934 24442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:54.170166 24442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:54.172523 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "8268d56ebe954329b20af5e9813a03dd"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "02a68904dfe0c164fcd77782a1918906"
server_key_iv: "08d6c114b059757eb29625218b3215cb"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.173117 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "8268d56ebe954329b20af5e9813a03dd"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "02a68904dfe0c164fcd77782a1918906"
server_key_iv: "08d6c114b059757eb29625218b3215cb"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.177074 24442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 14:39:54.179765 24457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.181063 24442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:39:54.181218 24442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "8268d56ebe954329b20af5e9813a03dd"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "02a68904dfe0c164fcd77782a1918906"
server_key_iv: "08d6c114b059757eb29625218b3215cb"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.181336 24442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:54.194813 24442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:54.197741 24442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:54.197944 24442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:54.198580 24442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:54.199468 24442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:54.199533 24442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.199618 24442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:54.199662 24442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.209141 24442 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:37805
I20260928 14:39:54.209157 24570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:37805 every 8 connection(s)
I20260928 14:39:54.210157 24442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:39:54.211515 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 24442
I20260928 14:39:54.211668 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:39:54.212029 6239 external_mini_cluster.cc:1428] Setting key 288ca32ef5caeb4ed6fd5da88bbba32c
Sep 28 14:39:54 dist-test-slave-06lh krb5kdc[24335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606394, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:39:54.226917 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.212144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:37531 (local address 127.6.23.254:35287)
0928 14:39:54.212284 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:54.212288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:54.213411 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 14:39:54.215337 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:54.215342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:54.215402 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:54.215493 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:54.217243 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.218141 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.219589 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.220093 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.220235 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.222883 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:54.222890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:54.222894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:54.222935 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:54.224428 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:54.225056 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:54.225059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:54.225060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:54.225177 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:54.225576 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:54.225583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:54.225588 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:54.225824 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:54.225907 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:54.226650 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 14:39:54.226785 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:39:54.227969 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.212668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35287 (local address 127.6.23.193:37531)
0928 14:39:54.213238 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:54.213298 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:54.215044 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:54.215696 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:54.215708 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:54.216283 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:54.217028 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.217052 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.218318 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.218321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.219447 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.219458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.220297 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.220300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.220464 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.221109 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:54.221127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:54.222641 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:54.224586 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:54.224595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:54.224638 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:54.224937 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:54.225282 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:54.225285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:54.225286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:54.225479 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:54.225935 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:54.225946 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:54.226326 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 14:39:54.226561 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 14:39:54.226971 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260928 14:39:54.229295 24571 heartbeater.cc:347] Connected to a master server at 127.6.23.254:35287
I20260928 14:39:54.229553 24571 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:54.230069 24571 heartbeater.cc:511] Master 127.6.23.254:35287 requested a full tablet report, sending...
I20260928 14:39:54.231882 24380 ts_manager.cc:194] Registered new tserver with Master: 8268d56ebe954329b20af5e9813a03dd (127.6.23.193:37805)
I20260928 14:39:54.233613 24380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:37531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:39:54.242759 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35287
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:54.356336 24576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:54.356685 24576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:54.356735 24576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:54.361004 24576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:54.361076 24576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:54.361137 24576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:39:54.366564 24576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:54.367687 24576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:54.368781 24576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:54.378528 24581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:54.378528 24584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:54.378531 24582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:54.378785 24576 server_base.cc:1034] running on GCE node
I20260928 14:39:54.379395 24576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:54.380056 24576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:54.381239 24576 hybrid_clock.cc:648] HybridClock initialized: now 1790606394381232 us; error 40 us; skew 500 ppm
Sep 28 14:39:54 dist-test-slave-06lh krb5kdc[24335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606394, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:54.384032 24576 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:39:54.384996 24576 webserver.cc:460] Webserver started at http://127.6.23.194:37383/ using document root <none> and password file <none>
I20260928 14:39:54.385612 24576 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:54.385697 24576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:54.385926 24576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:54.388068 24576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "fad21ad4482f459fb56687d10e2d2972"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "7e43f6664f3a94047ba6649da303586d"
server_key_iv: "c4570ad8b0b098f8736cf5f863411ae0"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.388613 24576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "fad21ad4482f459fb56687d10e2d2972"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "7e43f6664f3a94047ba6649da303586d"
server_key_iv: "c4570ad8b0b098f8736cf5f863411ae0"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.392360 24576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 14:39:54.394731 24591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.396009 24576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:54.396129 24576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "fad21ad4482f459fb56687d10e2d2972"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "7e43f6664f3a94047ba6649da303586d"
server_key_iv: "c4570ad8b0b098f8736cf5f863411ae0"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.396236 24576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:54.406515 24576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:54.409617 24576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:54.409826 24576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:54.410504 24576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:54.411379 24576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:54.411427 24576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.411489 24576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:54.411527 24576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.420372 24576 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:46255
I20260928 14:39:54.420414 24704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:46255 every 8 connection(s)
I20260928 14:39:54.421304 24576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:39:54.429564 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 24576
I20260928 14:39:54.429723 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:39:54.430089 6239 external_mini_cluster.cc:1428] Setting key 5469dc4c6510be2e518c4eb789297247
Sep 28 14:39:54 dist-test-slave-06lh krb5kdc[24335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606394, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:54.435787 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.423158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:42811 (local address 127.6.23.254:35287)
0928 14:39:54.423324 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:54.423328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:54.424205 (+ 877us) server_negotiation.cc:407] Connection header received
0928 14:39:54.425530 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:54.425534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:54.425594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:54.425696 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:54.426892 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.427511 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.428655 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.428884 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.428994 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.432038 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:54.432042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:54.432049 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:54.432072 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:54.433568 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:54.434106 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:54.434110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:54.434112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:54.434190 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:54.434580 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:54.434583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:54.434585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:54.434783 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:54.434866 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:54.435536 (+ 670us) server_negotiation.cc:299] Negotiation successful
0928 14:39:54.435642 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:39:54.437090 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.423588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35287 (local address 127.6.23.194:42811)
0928 14:39:54.424052 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:54.424095 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:54.425316 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:54.425882 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:54.425889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:54.426256 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:54.426746 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.426761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.427632 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.427635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.428528 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.428537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.429035 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.429039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.429208 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.429965 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:54.429988 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:54.431878 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:54.433703 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:54.433709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:54.433738 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:54.434002 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:54.434287 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:54.434291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:54.434293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:54.434457 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:54.434892 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:54.434902 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:54.435275 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 14:39:54.435504 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 14:39:54.435899 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":105,"threads_started":1}
I20260928 14:39:54.438364 24705 heartbeater.cc:347] Connected to a master server at 127.6.23.254:35287
I20260928 14:39:54.438613 24705 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:54.439110 24705 heartbeater.cc:511] Master 127.6.23.254:35287 requested a full tablet report, sending...
I20260928 14:39:54.440258 24380 ts_manager.cc:194] Registered new tserver with Master: fad21ad4482f459fb56687d10e2d2972 (127.6.23.194:46255)
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
I20260928 14:39:54.440840 24380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:42811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:39:54.458973 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35287
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:54.574055 24710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:54.574402 24710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:54.574497 24710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:54.578678 24710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:54.578775 24710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:54.578866 24710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:39:54.584352 24710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:54.585547 24710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:54.586655 24710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:54.596611 24715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:54.596736 24710 server_base.cc:1034] running on GCE node
W20260928 14:39:54.596611 24718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:54.596621 24716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:54.597424 24710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:54.598119 24710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:54.599289 24710 hybrid_clock.cc:648] HybridClock initialized: now 1790606394599254 us; error 48 us; skew 500 ppm
Sep 28 14:39:54 dist-test-slave-06lh krb5kdc[24335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606394, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:54.602259 24710 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:39:54.603336 24710 webserver.cc:460] Webserver started at http://127.6.23.195:35163/ using document root <none> and password file <none>
I20260928 14:39:54.603976 24710 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:54.604066 24710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:54.604291 24710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:54.606463 24710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "52a1d1e9934e4d8c832c7f642fc95a01"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "bc05dc8d45b1192da076c27a17acb819"
server_key_iv: "423ff8ff1c71290bbd9b6a64d28bd107"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.607059 24710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "52a1d1e9934e4d8c832c7f642fc95a01"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "bc05dc8d45b1192da076c27a17acb819"
server_key_iv: "423ff8ff1c71290bbd9b6a64d28bd107"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.610917 24710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 14:39:54.613596 24725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.614933 24710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:54.615064 24710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "52a1d1e9934e4d8c832c7f642fc95a01"
format_stamp: "Formatted at 2026-09-28 14:39:54 on dist-test-slave-06lh"
server_key: "bc05dc8d45b1192da076c27a17acb819"
server_key_iv: "423ff8ff1c71290bbd9b6a64d28bd107"
server_key_version: "encryptionkey@0"
I20260928 14:39:54.615178 24710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:54.638036 24710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:54.641232 24710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:54.641448 24710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:54.642053 24710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:54.642927 24710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:54.642994 24710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.643065 24710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:54.643105 24710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:54.652171 24710 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:44263
I20260928 14:39:54.652194 24838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:44263 every 8 connection(s)
I20260928 14:39:54.653077 24710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:39:54.655560 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 24710
I20260928 14:39:54.655714 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:39:54.656075 6239 external_mini_cluster.cc:1428] Setting key 962ff6a76f9b33078a5ce8503d869233
Sep 28 14:39:54 dist-test-slave-06lh krb5kdc[24335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606394, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:54.667445 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.654918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:53705 (local address 127.6.23.254:35287)
0928 14:39:54.655051 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:54.655055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:54.656106 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 14:39:54.657827 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:54.657830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:54.657887 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:54.657988 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:54.659288 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.659900 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.661050 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.661276 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.661385 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.663896 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:54.663900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:54.663903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:54.663936 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:54.665338 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:54.665904 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:54.665908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:54.665910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:54.665988 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:54.666314 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:54.666317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:54.666319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:54.666494 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:54.666596 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:54.667100 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 14:39:54.667267 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:39:54.668238 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.655397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35287 (local address 127.6.23.195:53705)
0928 14:39:54.655930 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:54.655973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:54.657621 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:54.658181 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:54.658188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:54.658614 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:54.659124 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.659139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.660022 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.660026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.660928 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.660937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.661428 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.661432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.661618 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.662226 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:54.662242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:54.663734 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:54.665468 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:54.665473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:54.665489 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:54.665801 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:54.666073 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:54.666078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:54.666080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:54.666217 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:54.666594 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:54.666601 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:54.666861 (+ 260us) client_negotiation.cc:769] Sending connection context
0928 14:39:54.667041 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 14:39:54.667416 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20260928 14:39:54.669463 24839 heartbeater.cc:347] Connected to a master server at 127.6.23.254:35287
I20260928 14:39:54.669711 24839 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:54.670210 24839 heartbeater.cc:511] Master 127.6.23.254:35287 requested a full tablet report, sending...
I20260928 14:39:54.671201 24379 ts_manager.cc:194] Registered new tserver with Master: 52a1d1e9934e4d8c832c7f642fc95a01 (127.6.23.195:44263)
I20260928 14:39:54.671852 24379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:53705
I20260928 14:39:54.682174 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:39:54.692699 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.684711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.6.23.254:35287)
0928 14:39:54.684892 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:54.684896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:54.685004 (+ 108us) server_negotiation.cc:407] Connection header received
0928 14:39:54.685189 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:54.685193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:54.685240 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:54.685329 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:54.686262 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.686787 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.687738 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.687975 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.688084 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.689609 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:54.689614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:54.689617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:54.689643 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:54.690851 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:54.691400 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:54.691403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:54.691405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:54.691489 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:54.691799 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:54.691809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:54.691811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:54.692019 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:54.692175 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:54.692436 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 14:39:54.692534 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:39:54.695549 24379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:39:54.698547 24379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:39:54.713465 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.707985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39646 (local address 127.6.23.194:46255)
0928 14:39:54.708493 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:54.708505 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:54.708527 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:39:54.708597 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:54.708601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:54.708772 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:54.709184 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:54.709354 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.710413 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.712133 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.712778 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.712889 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.713048 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:54.713215 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:39:54.713310 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":114,"threads_started":1}
I20260928 14:39:54.713457 24852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.707277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:46255 (local address 127.0.0.1:39646)
0928 14:39:54.708135 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:54.708148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:54.708229 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:54.708927 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:54.708933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:54.708955 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:54.709043 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.709049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.710547 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.710550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.712018 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.712025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.712905 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.712909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.713020 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.713064 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:39:54.713177 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:39:54.713305 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":81,"threads_started":1}
I20260928 14:39:54.713771 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.707294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.6.23.193:37805)
0928 14:39:54.707957 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:54.707974 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:54.707998 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:39:54.708084 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:54.708089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:54.708321 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:54.708482 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:54.709017 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.710219 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.711999 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.712810 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.712928 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.713073 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:54.713523 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 14:39:54.713622 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":66,"threads_started":1}
I20260928 14:39:54.713830 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.707166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:37805 (local address 127.0.0.1:50392)
0928 14:39:54.707530 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:54.707583 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:54.707716 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:54.708684 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:54.708690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:54.708713 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:54.708883 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.708893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.710362 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.710364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.711869 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.711879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.713136 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.713180 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.713263 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.713283 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:39:54.713347 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 14:39:54.713405 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":49,"thread_start_us":107,"threads_started":1}
I20260928 14:39:54.716045 24853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.707900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:44263 (local address 127.0.0.1:59414)
0928 14:39:54.708763 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:54.708777 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:54.708866 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:54.710993 (+ 2127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:54.711022 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:54.711040 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:54.711311 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.711319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.712692 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.712702 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.713849 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:54.713864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.715642 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.715646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:54.715742 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.715764 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:39:54.715830 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:39:54.715910 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":82,"threads_started":1}
I20260928 14:39:54.716089 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:54.708293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.6.23.195:44263)
0928 14:39:54.710528 (+ 2235us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:54.710543 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:54.710571 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:39:54.710631 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:54.710635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:54.710763 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:54.711024 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:54.711446 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.712540 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.714740 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:54.715518 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:54.715645 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:54.715800 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:54.715855 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 14:39:54.715948 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2116,"thread_start_us":159,"threads_started":1}
I20260928 14:39:54.717252 24639 tablet_service.cc:1467] Processing CreateTablet for tablet 42bb490487a940578c133a69764974ac (DEFAULT_TABLE table=test-table [id=c531c8330bd24879be4f05b6d89e2f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:54.718477 24639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bb490487a940578c133a69764974ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:54.718930 24505 tablet_service.cc:1467] Processing CreateTablet for tablet 42bb490487a940578c133a69764974ac (DEFAULT_TABLE table=test-table [id=c531c8330bd24879be4f05b6d89e2f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:54.718935 24773 tablet_service.cc:1467] Processing CreateTablet for tablet 42bb490487a940578c133a69764974ac (DEFAULT_TABLE table=test-table [id=c531c8330bd24879be4f05b6d89e2f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:54.719928 24505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bb490487a940578c133a69764974ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:54.719990 24773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bb490487a940578c133a69764974ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:54.725626 24856 tablet_bootstrap.cc:492] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd: Bootstrap starting.
I20260928 14:39:54.726969 24857 tablet_bootstrap.cc:492] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Bootstrap starting.
I20260928 14:39:54.727028 24858 tablet_bootstrap.cc:492] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Bootstrap starting.
I20260928 14:39:54.728140 24856 tablet_bootstrap.cc:654] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:54.729146 24858 tablet_bootstrap.cc:654] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:54.729387 24857 tablet_bootstrap.cc:654] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:54.729491 24856 log.cc:824] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:54.730340 24858 log.cc:824] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:54.730499 24857 log.cc:824] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:54.731700 24856 tablet_bootstrap.cc:492] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd: No bootstrap required, opened a new log
I20260928 14:39:54.731904 24856 ts_tablet_manager.cc:1397] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:39:54.732254 24857 tablet_bootstrap.cc:492] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: No bootstrap required, opened a new log
I20260928 14:39:54.732316 24858 tablet_bootstrap.cc:492] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: No bootstrap required, opened a new log
I20260928 14:39:54.732450 24857 ts_tablet_manager.cc:1397] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:39:54.732487 24858 ts_tablet_manager.cc:1397] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:39:54.735611 24857 raft_consensus.cc:348] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:54.735838 24857 raft_consensus.cc:374] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:54.735872 24857 raft_consensus.cc:729] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad21ad4482f459fb56687d10e2d2972, State: Initialized, Role: FOLLOWER
I20260928 14:39:54.735806 24856 raft_consensus.cc:348] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:54.736078 24856 raft_consensus.cc:374] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:54.736131 24856 raft_consensus.cc:729] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8268d56ebe954329b20af5e9813a03dd, State: Initialized, Role: FOLLOWER
I20260928 14:39:54.736104 24858 raft_consensus.cc:348] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:54.736330 24858 raft_consensus.cc:374] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:54.736274 24857 consensus_queue.cc:260] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:54.736377 24858 raft_consensus.cc:729] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a1d1e9934e4d8c832c7f642fc95a01, State: Initialized, Role: FOLLOWER
I20260928 14:39:54.736567 24856 consensus_queue.cc:260] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:54.736743 24858 consensus_queue.cc:260] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:54.737244 24571 heartbeater.cc:503] Master 127.6.23.254:35287 was elected leader, sending a full tablet report...
I20260928 14:39:54.737257 24857 ts_tablet_manager.cc:1428] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:39:54.737545 24705 heartbeater.cc:503] Master 127.6.23.254:35287 was elected leader, sending a full tablet report...
I20260928 14:39:54.737588 24858 ts_tablet_manager.cc:1428] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:39:54.737910 24839 heartbeater.cc:503] Master 127.6.23.254:35287 was elected leader, sending a full tablet report...
I20260928 14:39:54.738644 24856 ts_tablet_manager.cc:1428] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260928 14:39:54.904042 24840 tablet.cc:2369] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:39:54.922484 24706 tablet.cc:2369] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:39:54.961560 24572 tablet.cc:2369] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:39:55.021588 24863 raft_consensus.cc:482] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:55.021878 24863 raft_consensus.cc:504] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:55.023355 24863 leader_election.cc:288] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fad21ad4482f459fb56687d10e2d2972 (127.6.23.194:46255), 52a1d1e9934e4d8c832c7f642fc95a01 (127.6.23.195:44263)
I20260928 14:39:55.027439 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.023890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:57175 (local address 127.6.23.194:46255)
0928 14:39:55.024022 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:55.024027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:55.024044 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:39:55.024192 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:55.024198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:55.024261 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:55.024345 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:55.024682 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.025230 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.026434 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.026823 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.026917 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.026991 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:55.027204 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 14:39:55.027303 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:39:55.027467 24866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.023637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:44263 (local address 127.6.23.193:45501)
0928 14:39:55.024153 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:55.024167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:55.024273 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:55.024533 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:55.024536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:55.024550 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:55.024687 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.024693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.025393 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.025395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.026320 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.026329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.027010 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.027016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.027106 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.027123 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 14:39:55.027250 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 14:39:55.027314 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":62,"thread_start_us":111,"threads_started":1}
I20260928 14:39:55.027477 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.023740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:45501 (local address 127.6.23.195:44263)
0928 14:39:55.023963 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:55.023967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:55.024262 (+ 295us) server_negotiation.cc:407] Connection header received
0928 14:39:55.024379 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:55.024382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:55.024435 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:55.024516 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:55.024800 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.025268 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.026468 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.026862 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.027000 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.027052 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:55.027285 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 14:39:55.027370 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260928 14:39:55.027472 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.023576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:46255 (local address 127.6.23.193:57175)
0928 14:39:55.023946 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:55.023961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:55.024069 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:55.024365 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:55.024368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:55.024387 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:55.024543 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.024550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.025342 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.025347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.026306 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.026316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.026959 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.026962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.027055 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.027086 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:39:55.027171 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:39:55.027279 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":55,"thread_start_us":159,"threads_started":1}
I20260928 14:39:55.028323 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb490487a940578c133a69764974ac" candidate_uuid: "8268d56ebe954329b20af5e9813a03dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad21ad4482f459fb56687d10e2d2972" is_pre_election: true
I20260928 14:39:55.028326 24793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb490487a940578c133a69764974ac" candidate_uuid: "8268d56ebe954329b20af5e9813a03dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" is_pre_election: true
I20260928 14:39:55.028649 24793 raft_consensus.cc:2454] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8268d56ebe954329b20af5e9813a03dd in term 0.
I20260928 14:39:55.028649 24659 raft_consensus.cc:2454] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8268d56ebe954329b20af5e9813a03dd in term 0.
I20260928 14:39:55.029186 24459 leader_election.cc:302] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52a1d1e9934e4d8c832c7f642fc95a01, 8268d56ebe954329b20af5e9813a03dd, fad21ad4482f459fb56687d10e2d2972; no voters:
I20260928 14:39:55.029495 24863 raft_consensus.cc:2781] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:39:55.029595 24863 raft_consensus.cc:482] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:39:55.029685 24863 raft_consensus.cc:3037] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:55.030774 24863 raft_consensus.cc:504] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:55.031121 24863 leader_election.cc:288] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [CANDIDATE]: Term 1 election: Requested vote from peers fad21ad4482f459fb56687d10e2d2972 (127.6.23.194:46255), 52a1d1e9934e4d8c832c7f642fc95a01 (127.6.23.195:44263)
I20260928 14:39:55.031536 24793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb490487a940578c133a69764974ac" candidate_uuid: "8268d56ebe954329b20af5e9813a03dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a1d1e9934e4d8c832c7f642fc95a01"
I20260928 14:39:55.031539 24659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bb490487a940578c133a69764974ac" candidate_uuid: "8268d56ebe954329b20af5e9813a03dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad21ad4482f459fb56687d10e2d2972"
I20260928 14:39:55.031677 24793 raft_consensus.cc:3037] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:55.031679 24659 raft_consensus.cc:3037] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:55.032703 24793 raft_consensus.cc:2454] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8268d56ebe954329b20af5e9813a03dd in term 1.
I20260928 14:39:55.032753 24659 raft_consensus.cc:2454] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8268d56ebe954329b20af5e9813a03dd in term 1.
I20260928 14:39:55.033061 24461 leader_election.cc:302] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a1d1e9934e4d8c832c7f642fc95a01, 8268d56ebe954329b20af5e9813a03dd; no voters:
I20260928 14:39:55.033274 24863 raft_consensus.cc:2781] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:55.033512 24863 raft_consensus.cc:686] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 1 LEADER]: Becoming Leader. State: Replica: 8268d56ebe954329b20af5e9813a03dd, State: Running, Role: LEADER
I20260928 14:39:55.033842 24863 consensus_queue.cc:237] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } }
I20260928 14:39:55.037087 24377 catalog_manager.cc:5495] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8268d56ebe954329b20af5e9813a03dd (127.6.23.193). New cstate: current_term: 1 leader_uuid: "8268d56ebe954329b20af5e9813a03dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8268d56ebe954329b20af5e9813a03dd" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 37805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 } health_report { overall_health: UNKNOWN } } }
I20260928 14:39:55.082348 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.079080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.6.23.193:37805)
0928 14:39:55.079204 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:55.079207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:55.079222 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:39:55.079394 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:55.079396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:55.079454 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:55.079548 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:39:55.079772 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.080268 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.081061 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.081282 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.081358 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.081608 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:39:55.082020 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:39:55.082121 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:55.082202 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:39:55.082236 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:39:55.088362 24659 raft_consensus.cc:1261] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 1 FOLLOWER]: Refusing update from remote peer 8268d56ebe954329b20af5e9813a03dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:55.088370 24793 raft_consensus.cc:1261] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 1 FOLLOWER]: Refusing update from remote peer 8268d56ebe954329b20af5e9813a03dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:55.089044 24867 consensus_queue.cc:1035] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a1d1e9934e4d8c832c7f642fc95a01" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 44263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:55.089272 24863 consensus_queue.cc:1035] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fad21ad4482f459fb56687d10e2d2972" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 46255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:55.097808 24871 mvcc.cc:205] Tried to move back new op lower bound from 7334323794274623488 to 7334323794061008896. Current Snapshot: MvccSnapshot[applied={T|T < 7334323794274623488}]
I20260928 14:39:55.105083 24377 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:55.105234 24377 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:55.107795 24377 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4f73378fd7d40ecb8df042f88c71965: Sending DeleteTablet for 3 replicas of tablet 42bb490487a940578c133a69764974ac
I20260928 14:39:55.108551 24505 tablet_service.cc:1514] Processing DeleteTablet for tablet 42bb490487a940578c133a69764974ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:50392
I20260928 14:39:55.108585 24639 tablet_service.cc:1514] Processing DeleteTablet for tablet 42bb490487a940578c133a69764974ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:39646
I20260928 14:39:55.108677 24773 tablet_service.cc:1514] Processing DeleteTablet for tablet 42bb490487a940578c133a69764974ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:55 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:59414
I20260928 14:39:55.109256 24879 tablet_replica.cc:330] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd: stopping tablet replica
I20260928 14:39:55.109577 24879 raft_consensus.cc:2229] T 42bb490487a940578c133a69764974ac P 8268d56ebe954329b20af5e9813a03dd [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:39:55.109758 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 24442
I20260928 14:39:55.111378 24880 tablet_replica.cc:330] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: stopping tablet replica
I20260928 14:39:55.111671 24880 raft_consensus.cc:2229] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:55.111997 24880 raft_consensus.cc:2258] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:55.112365 24881 tablet_replica.cc:330] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: stopping tablet replica
I20260928 14:39:55.112636 24881 raft_consensus.cc:2229] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:55.112911 24881 raft_consensus.cc:2258] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:55.113617 24880 ts_tablet_manager.cc:1905] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:55.114259 24881 ts_tablet_manager.cc:1905] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:55.116077 24880 ts_tablet_manager.cc:1918] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:55.116187 24880 log.cc:1196] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/42bb490487a940578c133a69764974ac
I20260928 14:39:55.116459 24880 ts_tablet_manager.cc:1939] T 42bb490487a940578c133a69764974ac P fad21ad4482f459fb56687d10e2d2972: Deleting consensus metadata
I20260928 14:39:55.116968 24881 ts_tablet_manager.cc:1918] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:55.117076 24881 log.cc:1196] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/42bb490487a940578c133a69764974ac
W20260928 14:39:55.117224 24364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:37805 (error 108)
I20260928 14:39:55.117403 24881 ts_tablet_manager.cc:1939] T 42bb490487a940578c133a69764974ac P 52a1d1e9934e4d8c832c7f642fc95a01: Deleting consensus metadata
I20260928 14:39:55.117810 24365 catalog_manager.cc:4841] TS fad21ad4482f459fb56687d10e2d2972 (127.6.23.194:46255): tablet 42bb490487a940578c133a69764974ac (table test-table [id=c531c8330bd24879be4f05b6d89e2f52]) successfully deleted
I20260928 14:39:55.118323 24853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.117997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:37805 (local address 127.0.0.1:50410)
0928 14:39:55.118174 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:55.118239 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:37805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260928 14:39:55.118492 24364 catalog_manager.cc:4584] TS 8268d56ebe954329b20af5e9813a03dd (127.6.23.193:37805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42bb490487a940578c133a69764974ac: Network error: Client connection negotiation failed: client connection to 127.6.23.193:37805: connect: Connection refused (error 111)
I20260928 14:39:55.118762 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 24576
I20260928 14:39:55.118794 24367 catalog_manager.cc:4841] TS 52a1d1e9934e4d8c832c7f642fc95a01 (127.6.23.195:44263): tablet 42bb490487a940578c133a69764974ac (table test-table [id=c531c8330bd24879be4f05b6d89e2f52]) successfully deleted
I20260928 14:39:55.125152 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 24710
I20260928 14:39:55.131145 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 24348
2026-09-28T14:39:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2539 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T14:39:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:39:55Z Disabled control of system clock
I20260928 14:39:55.162626 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:33887
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:33887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:39:55.280344 24886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:55.280633 24886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:55.280671 24886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:55.285012 24886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:39:55.285084 24886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:55.285099 24886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:55.285117 24886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:39:55.285179 24886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:39:55.290654 24886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:33887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:33887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:55.292380 24886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:55.293934 24886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:55.302587 24894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:55.302700 24891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:55.302917 24886 server_base.cc:1034] running on GCE node
W20260928 14:39:55.303088 24892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:55.303711 24886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:55.304879 24886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:55.306072 24886 hybrid_clock.cc:648] HybridClock initialized: now 1790606395306043 us; error 41 us; skew 500 ppm
I20260928 14:39:55.308310 24886 webserver.cc:460] Webserver started at http://127.6.23.254:44273/ using document root <none> and password file <none>
I20260928 14:39:55.308943 24886 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:55.309015 24886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:55.309266 24886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:55.311448 24886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "82b2a6e0dd5244cba388658ece3f676a"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "085801fc1eeed379689ecc39a2dcdaa1"
server_key_iv: "669c378439cae12cd7e4f0d05c9f01af"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.312058 24886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "82b2a6e0dd5244cba388658ece3f676a"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "085801fc1eeed379689ecc39a2dcdaa1"
server_key_iv: "669c378439cae12cd7e4f0d05c9f01af"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.316365 24886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:39:55.318910 24900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.320293 24886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:39:55.320454 24886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "82b2a6e0dd5244cba388658ece3f676a"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "085801fc1eeed379689ecc39a2dcdaa1"
server_key_iv: "669c378439cae12cd7e4f0d05c9f01af"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.320595 24886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:55.338585 24886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:55.339444 24886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:55.339649 24886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:55.346704 24886 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:33887
I20260928 14:39:55.346714 24952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:33887 every 8 connection(s)
I20260928 14:39:55.347731 24886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:39:55.349138 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 24886
I20260928 14:39:55.349267 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:39:55.349596 6239 external_mini_cluster.cc:1428] Setting key 22722bd634c4f95342b4e61388f6f08b
I20260928 14:39:55.351573 24953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:55.357230 24953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a: Bootstrap starting.
I20260928 14:39:55.359047 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.351045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.6.23.254:33887)
0928 14:39:55.351789 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:55.351805 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:55.351844 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:39:55.352510 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:55.352527 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:55.352924 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:55.353270 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:55.353738 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.354704 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.355700 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.356182 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.356370 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.357262 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:55.357267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:55.357280 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:55.357726 (+ 446us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:55.357792 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:55.357813 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:55.357927 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:55.358042 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 14:39:55.358313 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":184,"threads_started":1}
I20260928 14:39:55.359681 24953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:55.360445 24953 log.cc:824] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:55.362344 24953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a: No bootstrap required, opened a new log
I20260928 14:39:55.365917 24953 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b2a6e0dd5244cba388658ece3f676a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33887 } }
I20260928 14:39:55.366183 24953 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:55.366278 24953 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b2a6e0dd5244cba388658ece3f676a, State: Initialized, Role: FOLLOWER
I20260928 14:39:55.366701 24953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b2a6e0dd5244cba388658ece3f676a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33887 } }
I20260928 14:39:55.366856 24953 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:39:55.366945 24953 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:39:55.367061 24953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:55.368062 24953 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b2a6e0dd5244cba388658ece3f676a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33887 } }
I20260928 14:39:55.368397 24953 leader_election.cc:302] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82b2a6e0dd5244cba388658ece3f676a; no voters:
I20260928 14:39:55.368698 24953 leader_election.cc:288] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:39:55.368835 24958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:55.369084 24958 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [term 1 LEADER]: Becoming Leader. State: Replica: 82b2a6e0dd5244cba388658ece3f676a, State: Running, Role: LEADER
I20260928 14:39:55.369380 24958 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b2a6e0dd5244cba388658ece3f676a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33887 } }
I20260928 14:39:55.369916 24953 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:39:55.371186 24960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b2a6e0dd5244cba388658ece3f676a. Latest consensus state: current_term: 1 leader_uuid: "82b2a6e0dd5244cba388658ece3f676a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b2a6e0dd5244cba388658ece3f676a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33887 } } }
I20260928 14:39:55.371186 24959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82b2a6e0dd5244cba388658ece3f676a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82b2a6e0dd5244cba388658ece3f676a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 33887 } } }
I20260928 14:39:55.371384 24960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:55.371384 24959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:55.371784 24968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:39:55.374567 24968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:39:55.379848 24968 catalog_manager.cc:1347] Generated new cluster ID: 93aba55e9b08483796c7227232cbcab5
I20260928 14:39:55.379913 24968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:39:55.394928 24968 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:39:55.395949 24968 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:39:55.403402 24968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a: Generated new TSK 0
I20260928 14:39:55.404100 24968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:39:55.413769 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33887
--builtin_ntp_servers=127.6.23.212:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:39:55.527324 24977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:55.527606 24977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:55.527673 24977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:55.531791 24977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:55.531852 24977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:55.531947 24977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:39:55.536768 24977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:55.537950 24977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:55.539471 24977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:55.549017 24983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:55.549026 24982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:55.549026 24985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:55.549181 24977 server_base.cc:1034] running on GCE node
I20260928 14:39:55.549656 24977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:55.550319 24977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:55.551509 24977 hybrid_clock.cc:648] HybridClock initialized: now 1790606395551498 us; error 32 us; skew 500 ppm
I20260928 14:39:55.553381 24977 webserver.cc:460] Webserver started at http://127.6.23.193:46257/ using document root <none> and password file <none>
I20260928 14:39:55.553964 24977 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:55.554052 24977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:55.554276 24977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:55.556378 24977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "c1b4aca2f0eb49cf9a062a66466b84e7"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "fa152a64630195c1694066b9de5e8b29"
server_key_iv: "874f83a64225b090fa21fbb05b98d197"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.556948 24977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "c1b4aca2f0eb49cf9a062a66466b84e7"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "fa152a64630195c1694066b9de5e8b29"
server_key_iv: "874f83a64225b090fa21fbb05b98d197"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.560613 24977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:39:55.562960 24991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.564227 24977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:55.564389 24977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "c1b4aca2f0eb49cf9a062a66466b84e7"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "fa152a64630195c1694066b9de5e8b29"
server_key_iv: "874f83a64225b090fa21fbb05b98d197"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.564507 24977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:55.583618 24977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:55.584357 24977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:55.584558 24977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:55.585162 24977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:55.586114 24977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:55.586184 24977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.586257 24977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:55.586299 24977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.595455 24977 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:41067
I20260928 14:39:55.595482 25104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:41067 every 8 connection(s)
I20260928 14:39:55.596350 24977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:39:55.600298 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 24977
I20260928 14:39:55.600461 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:39:55.600845 6239 external_mini_cluster.cc:1428] Setting key d03f004e492bbfeb436a4c93f474a103
I20260928 14:39:55.602854 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33887
--builtin_ntp_servers=127.6.23.212:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:39:55.606345 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.598127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:42853 (local address 127.6.23.254:33887)
0928 14:39:55.598258 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:55.598266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:55.599375 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 14:39:55.601088 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:55.601091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:55.601144 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:55.601262 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:55.602288 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.602834 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.604147 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.604383 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.604466 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.605735 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:55.605739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:55.605742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:55.605760 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:55.605786 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:55.605800 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:55.605882 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:55.606018 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 14:39:55.606122 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:39:55.607079 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.598546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33887 (local address 127.6.23.193:42853)
0928 14:39:55.599172 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:55.599212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:55.600845 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:55.601422 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:55.601428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:55.601511 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:55.602151 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.602168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.602959 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.602965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.604021 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.604035 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.604537 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.604543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.604760 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.605443 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:55.605445 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:55.605510 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:55.605572 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:55.605888 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:55.605894 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:55.605898 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:39:55.605991 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:39:55.606206 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260928 14:39:55.608348 25105 heartbeater.cc:347] Connected to a master server at 127.6.23.254:33887
I20260928 14:39:55.608567 25105 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:55.609056 25105 heartbeater.cc:511] Master 127.6.23.254:33887 requested a full tablet report, sending...
I20260928 14:39:55.610468 24917 ts_manager.cc:194] Registered new tserver with Master: c1b4aca2f0eb49cf9a062a66466b84e7 (127.6.23.193:41067)
I20260928 14:39:55.611855 24917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.193:42853
W20260928 14:39:55.719935 25108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:55.720191 25108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:55.720229 25108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:55.724460 25108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:55.724514 25108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:55.724598 25108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:39:55.729466 25108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:55.730607 25108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:55.732076 25108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:55.743518 25113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:55.743520 25114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:55.743515 25116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:55.743918 25108 server_base.cc:1034] running on GCE node
I20260928 14:39:55.744334 25108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:55.745041 25108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:55.746215 25108 hybrid_clock.cc:648] HybridClock initialized: now 1790606395746190 us; error 46 us; skew 500 ppm
I20260928 14:39:55.748190 25108 webserver.cc:460] Webserver started at http://127.6.23.194:37563/ using document root <none> and password file <none>
I20260928 14:39:55.748777 25108 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:55.748838 25108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:55.749047 25108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:55.751142 25108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "a9c344c2283d4143b5a80ceb2ecc51b6"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "3a3728169ad040428bb5f5bcf0219a71"
server_key_iv: "563ea4fe785a8e4726d462ad004c094f"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.751765 25108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "a9c344c2283d4143b5a80ceb2ecc51b6"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "3a3728169ad040428bb5f5bcf0219a71"
server_key_iv: "563ea4fe785a8e4726d462ad004c094f"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.755614 25108 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:39:55.758181 25122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.759563 25108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:39:55.759692 25108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "a9c344c2283d4143b5a80ceb2ecc51b6"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "3a3728169ad040428bb5f5bcf0219a71"
server_key_iv: "563ea4fe785a8e4726d462ad004c094f"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.759807 25108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:55.775302 25108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:55.775961 25108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:55.776146 25108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:55.776764 25108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:55.777747 25108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:55.777814 25108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.777884 25108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:55.777935 25108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.788622 25108 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:41841
I20260928 14:39:55.788640 25235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:41841 every 8 connection(s)
I20260928 14:39:55.789536 25108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:39:55.790062 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 25108
I20260928 14:39:55.790200 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:39:55.790570 6239 external_mini_cluster.cc:1428] Setting key 101d023cb0fa6a68a19fdf96da0bb05b
I20260928 14:39:55.793385 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33887
--builtin_ntp_servers=127.6.23.212:40837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:39:55.801347 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.791469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:36305 (local address 127.6.23.254:33887)
0928 14:39:55.792919 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:55.792924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:55.792938 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:39:55.794366 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:55.794370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:55.794425 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:55.794504 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:55.795706 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.796566 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.798455 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.798905 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.799002 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.800649 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:55.800656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:55.800660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:55.800678 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:55.800705 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:55.800719 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:55.800785 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:55.801047 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 14:39:55.801134 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1368}
I20260928 14:39:55.803160 25238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.791963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33887 (local address 127.6.23.194:36305)
0928 14:39:55.792718 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:55.792782 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:55.794160 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:55.794673 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:55.794680 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:55.794772 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:55.795527 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.795550 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.796729 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.796746 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.798296 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.798311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.799242 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.799249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.799503 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.800285 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:55.800289 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:55.800353 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:55.800451 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:55.800860 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:55.800872 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:55.800879 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:39:55.801614 (+ 735us) client_negotiation.cc:240] Negotiation successful
0928 14:39:55.801970 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":128,"threads_started":1}
I20260928 14:39:55.804574 25236 heartbeater.cc:347] Connected to a master server at 127.6.23.254:33887
I20260928 14:39:55.804847 25236 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:55.805550 25236 heartbeater.cc:511] Master 127.6.23.254:33887 requested a full tablet report, sending...
I20260928 14:39:55.806677 24917 ts_manager.cc:194] Registered new tserver with Master: a9c344c2283d4143b5a80ceb2ecc51b6 (127.6.23.194:41841)
I20260928 14:39:55.807478 24917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.194:36305
W20260928 14:39:55.909358 25239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:55.909669 25239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:55.909734 25239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:55.913959 25239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:55.914016 25239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:55.914108 25239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:39:55.918888 25239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:33887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:55.920090 25239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:55.921506 25239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:55.931259 25247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:55.931299 25245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:55.931546 25239 server_base.cc:1034] running on GCE node
W20260928 14:39:55.931315 25244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:55.932111 25239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:55.932777 25239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:55.933938 25239 hybrid_clock.cc:648] HybridClock initialized: now 1790606395933918 us; error 33 us; skew 500 ppm
I20260928 14:39:55.935885 25239 webserver.cc:460] Webserver started at http://127.6.23.195:36011/ using document root <none> and password file <none>
I20260928 14:39:55.936483 25239 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:55.936545 25239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:55.936764 25239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:55.938810 25239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "01b50763f7a84ce1b25ece17c55f58d7"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "5108ea7b13341764046111bbf180156e"
server_key_iv: "084bce0aca4c72bb03c08063df4ff022"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.939416 25239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "01b50763f7a84ce1b25ece17c55f58d7"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "5108ea7b13341764046111bbf180156e"
server_key_iv: "084bce0aca4c72bb03c08063df4ff022"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.942962 25239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:39:55.945464 25253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.946678 25239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:39:55.946811 25239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "01b50763f7a84ce1b25ece17c55f58d7"
format_stamp: "Formatted at 2026-09-28 14:39:55 on dist-test-slave-06lh"
server_key: "5108ea7b13341764046111bbf180156e"
server_key_iv: "084bce0aca4c72bb03c08063df4ff022"
server_key_version: "encryptionkey@0"
I20260928 14:39:55.946924 25239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:55.979113 25239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:55.979864 25239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:55.980069 25239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:55.980691 25239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:55.981633 25239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:55.981704 25239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.981778 25239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:55.981824 25239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:55.990312 25239 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:37943
I20260928 14:39:55.990330 25366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:37943 every 8 connection(s)
I20260928 14:39:55.991194 25239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:39:56.000291 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.993030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:60769 (local address 127.6.23.254:33887)
0928 14:39:55.993168 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:55.993171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:55.994214 (+ 1043us) server_negotiation.cc:407] Connection header received
0928 14:39:55.995673 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:55.995677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:55.995736 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:55.995838 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:55.996695 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.997229 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.998361 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.998584 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.998695 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.999789 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:55.999792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:55.999795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:55.999815 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:55.999834 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:55.999842 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:55.999917 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:56.000077 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 14:39:56.000135 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:39:56.001078 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:55.993444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:33887 (local address 127.6.23.195:60769)
0928 14:39:55.994072 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:55.994112 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:55.995466 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:55.996011 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:55.996017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:55.996101 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:55.996561 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.996573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.997361 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.997369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.998237 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:55.998246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:55.998733 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:55.998737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:55.998907 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:55.999537 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:55.999539 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:55.999575 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:55.999634 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:55.999918 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:55.999930 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:55.999937 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:39:56.000063 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:39:56.000315 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260928 14:39:56.001082 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 25239
I20260928 14:39:56.001207 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:39:56.001533 6239 external_mini_cluster.cc:1428] Setting key 7b22c051391e3d4e2e4b3b91dbaa3f44
I20260928 14:39:56.002220 25367 heartbeater.cc:347] Connected to a master server at 127.6.23.254:33887
I20260928 14:39:56.002463 25367 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:56.003037 25367 heartbeater.cc:511] Master 127.6.23.254:33887 requested a full tablet report, sending...
I20260928 14:39:56.004074 24917 ts_manager.cc:194] Registered new tserver with Master: 01b50763f7a84ce1b25ece17c55f58d7 (127.6.23.195:37943)
I20260928 14:39:56.004639 24917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.195:60769
I20260928 14:39:56.015404 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:39:56.021381 24956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.017389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.6.23.254:33887)
0928 14:39:56.017553 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:56.017557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:56.017704 (+ 147us) server_negotiation.cc:407] Connection header received
0928 14:39:56.017859 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:56.017862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:56.017912 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:56.017997 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:56.018277 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.018808 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.019736 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.019961 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.020070 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.020866 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:56.020870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:56.020874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:56.020891 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:56.020908 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:56.020915 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:56.021040 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:56.021195 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 14:39:56.021230 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:39:56.024202 24917 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:39:56.027408 24917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:39:56.041718 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.035752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:41841 (local address 127.0.0.1:53220)
0928 14:39:56.036089 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:56.036142 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:56.036273 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:56.037535 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:56.037540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:56.037564 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:56.037711 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.037718 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.038753 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.038759 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.040155 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.040165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.041161 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.041165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.041281 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.041318 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:39:56.041435 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 14:39:56.041555 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":166,"thread_start_us":89,"threads_started":1}
I20260928 14:39:56.041723 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.035881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53220 (local address 127.6.23.194:41841)
0928 14:39:56.037057 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:56.037071 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:56.037101 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:39:56.037180 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:56.037185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:56.037382 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:56.037545 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:56.037838 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.038639 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.040283 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.041037 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.041162 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.041359 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:56.041463 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:39:56.041556 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":127,"threads_started":1}
I20260928 14:39:56.042153 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.036519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41067 (local address 127.0.0.1:56744)
0928 14:39:56.036779 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:56.036794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:56.036877 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:56.037701 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:56.037705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:56.037719 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:56.037829 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.037835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.039204 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.039207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.040455 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.040463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.041723 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.041726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.041857 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.041881 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:39:56.041943 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 14:39:56.042008 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":35,"thread_start_us":94,"threads_started":1}
I20260928 14:39:56.042336 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.035945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:37943 (local address 127.0.0.1:33138)
0928 14:39:56.036614 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:56.036630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:56.036719 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:56.037670 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:56.037673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:56.037690 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:56.037807 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.037815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.039354 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.039358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.040660 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.040670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.041692 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.041698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.042061 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.042082 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:39:56.042137 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 14:39:56.042212 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":204,"thread_start_us":79,"threads_started":1}
I20260928 14:39:56.042565 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.036938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.6.23.193:41067)
0928 14:39:56.037338 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:56.037348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:56.037368 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:39:56.037429 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:56.037433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:56.037576 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:56.037707 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:56.037939 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.039086 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.040597 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.041378 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.041485 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.041676 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:56.042330 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 14:39:56.042435 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":109,"threads_started":1}
I20260928 14:39:56.042955 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.036571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.6.23.195:37943)
0928 14:39:56.037168 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:56.037179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:56.037204 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:39:56.037311 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:56.037316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:56.037510 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:56.037983 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:56.038108 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.039183 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.040789 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.041561 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.042460 (+ 899us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.042644 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:56.042697 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:39:56.042806 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":166,"threads_started":1}
I20260928 14:39:56.045044 25039 tablet_service.cc:1467] Processing CreateTablet for tablet ef9d14c9724f4f15b85f23504f23c414 (DEFAULT_TABLE table=test-table [id=e02f122b695e419c9d80f366c3281d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:56.045218 25301 tablet_service.cc:1467] Processing CreateTablet for tablet ef9d14c9724f4f15b85f23504f23c414 (DEFAULT_TABLE table=test-table [id=e02f122b695e419c9d80f366c3281d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:56.045171 25170 tablet_service.cc:1467] Processing CreateTablet for tablet ef9d14c9724f4f15b85f23504f23c414 (DEFAULT_TABLE table=test-table [id=e02f122b695e419c9d80f366c3281d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:56.045929 25039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef9d14c9724f4f15b85f23504f23c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:56.046169 25170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef9d14c9724f4f15b85f23504f23c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:56.046169 25301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef9d14c9724f4f15b85f23504f23c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:56.052240 25384 tablet_bootstrap.cc:492] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7: Bootstrap starting.
I20260928 14:39:56.053539 25385 tablet_bootstrap.cc:492] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: Bootstrap starting.
I20260928 14:39:56.054388 25386 tablet_bootstrap.cc:492] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: Bootstrap starting.
I20260928 14:39:56.054679 25384 tablet_bootstrap.cc:654] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:56.055714 25385 tablet_bootstrap.cc:654] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:56.056151 25384 log.cc:824] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:56.056201 25386 tablet_bootstrap.cc:654] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:56.056524 25385 log.cc:824] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:56.056988 25386 log.cc:824] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:56.058226 25384 tablet_bootstrap.cc:492] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7: No bootstrap required, opened a new log
I20260928 14:39:56.058425 25384 ts_tablet_manager.cc:1397] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:39:56.058903 25386 tablet_bootstrap.cc:492] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: No bootstrap required, opened a new log
I20260928 14:39:56.059010 25385 tablet_bootstrap.cc:492] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: No bootstrap required, opened a new log
I20260928 14:39:56.059073 25386 ts_tablet_manager.cc:1397] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:39:56.059186 25385 ts_tablet_manager.cc:1397] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:39:56.062170 25385 raft_consensus.cc:348] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.062443 25385 raft_consensus.cc:374] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:56.062496 25385 raft_consensus.cc:729] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b50763f7a84ce1b25ece17c55f58d7, State: Initialized, Role: FOLLOWER
I20260928 14:39:56.062470 25384 raft_consensus.cc:348] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.062826 25384 raft_consensus.cc:374] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:56.062887 25384 raft_consensus.cc:729] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b4aca2f0eb49cf9a062a66466b84e7, State: Initialized, Role: FOLLOWER
I20260928 14:39:56.062963 25385 consensus_queue.cc:260] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.063338 25384 consensus_queue.cc:260] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.063212 25386 raft_consensus.cc:348] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.063476 25386 raft_consensus.cc:374] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:56.063529 25386 raft_consensus.cc:729] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9c344c2283d4143b5a80ceb2ecc51b6, State: Initialized, Role: FOLLOWER
I20260928 14:39:56.063934 25386 consensus_queue.cc:260] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.064028 25385 ts_tablet_manager.cc:1428] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260928 14:39:56.064324 25384 ts_tablet_manager.cc:1428] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:39:56.064383 25367 heartbeater.cc:503] Master 127.6.23.254:33887 was elected leader, sending a full tablet report...
I20260928 14:39:56.064635 25105 heartbeater.cc:503] Master 127.6.23.254:33887 was elected leader, sending a full tablet report...
I20260928 14:39:56.064793 25386 ts_tablet_manager.cc:1428] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 14:39:56.065063 25236 heartbeater.cc:503] Master 127.6.23.254:33887 was elected leader, sending a full tablet report...
I20260928 14:39:56.077946 25390 raft_consensus.cc:482] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:56.078107 25390 raft_consensus.cc:504] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.079139 25390 leader_election.cc:288] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9c344c2283d4143b5a80ceb2ecc51b6 (127.6.23.194:41841), c1b4aca2f0eb49cf9a062a66466b84e7 (127.6.23.193:41067)
I20260928 14:39:56.082527 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.079422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:44237 (local address 127.6.23.194:41841)
0928 14:39:56.079548 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:56.079552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:56.079582 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:39:56.079668 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:56.079671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:56.079720 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:56.079803 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:56.080084 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.080585 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.081604 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.082006 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.082129 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.082177 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:56.082351 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:39:56.082412 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:39:56.082527 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.079336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:41841 (local address 127.6.23.195:44237)
0928 14:39:56.079465 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:56.079489 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:56.079576 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:56.079826 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:56.079829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:56.079846 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:56.079967 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.079973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.080690 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.080696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.081478 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.081487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.082123 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.082127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.082224 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.082250 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:39:56.082336 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:39:56.082405 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":5}
I20260928 14:39:56.083097 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.079716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:48641 (local address 127.6.23.193:41067)
0928 14:39:56.079853 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:56.079858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:56.080119 (+ 261us) server_negotiation.cc:407] Connection header received
0928 14:39:56.080259 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:56.080262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:56.080310 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:56.080385 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:56.080652 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.081132 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.082126 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.082511 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.082645 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.082688 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:56.082899 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 14:39:56.082973 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:39:56.083348 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9d14c9724f4f15b85f23504f23c414" candidate_uuid: "01b50763f7a84ce1b25ece17c55f58d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" is_pre_election: true
I20260928 14:39:56.083477 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.079617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41067 (local address 127.6.23.195:48641)
0928 14:39:56.080024 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:56.080035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:56.080147 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:56.080405 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:56.080408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:56.080423 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:56.080542 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.080547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.081238 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.081252 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.082000 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:56.082008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.082653 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.082657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:56.082757 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.082781 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:39:56.083270 (+ 489us) client_negotiation.cc:240] Negotiation successful
0928 14:39:56.083348 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":163,"threads_started":1}
I20260928 14:39:56.083681 25190 raft_consensus.cc:2454] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b50763f7a84ce1b25ece17c55f58d7 in term 0.
I20260928 14:39:56.084031 25059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9d14c9724f4f15b85f23504f23c414" candidate_uuid: "01b50763f7a84ce1b25ece17c55f58d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" is_pre_election: true
I20260928 14:39:56.084189 25256 leader_election.cc:302] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b50763f7a84ce1b25ece17c55f58d7, a9c344c2283d4143b5a80ceb2ecc51b6; no voters:
I20260928 14:39:56.084316 25059 raft_consensus.cc:2454] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01b50763f7a84ce1b25ece17c55f58d7 in term 0.
I20260928 14:39:56.084504 25390 raft_consensus.cc:2781] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:39:56.084595 25390 raft_consensus.cc:482] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:39:56.084651 25390 raft_consensus.cc:3037] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:56.085636 25390 raft_consensus.cc:504] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.085958 25390 leader_election.cc:288] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [CANDIDATE]: Term 1 election: Requested vote from peers a9c344c2283d4143b5a80ceb2ecc51b6 (127.6.23.194:41841), c1b4aca2f0eb49cf9a062a66466b84e7 (127.6.23.193:41067)
I20260928 14:39:56.086310 25190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9d14c9724f4f15b85f23504f23c414" candidate_uuid: "01b50763f7a84ce1b25ece17c55f58d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6"
I20260928 14:39:56.086404 25059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef9d14c9724f4f15b85f23504f23c414" candidate_uuid: "01b50763f7a84ce1b25ece17c55f58d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7"
I20260928 14:39:56.086452 25190 raft_consensus.cc:3037] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:56.086551 25059 raft_consensus.cc:3037] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:56.087527 25190 raft_consensus.cc:2454] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b50763f7a84ce1b25ece17c55f58d7 in term 1.
I20260928 14:39:56.087715 25059 raft_consensus.cc:2454] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01b50763f7a84ce1b25ece17c55f58d7 in term 1.
I20260928 14:39:56.087821 25256 leader_election.cc:302] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01b50763f7a84ce1b25ece17c55f58d7, a9c344c2283d4143b5a80ceb2ecc51b6; no voters:
I20260928 14:39:56.087977 25390 raft_consensus.cc:2781] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:56.088188 25390 raft_consensus.cc:686] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 1 LEADER]: Becoming Leader. State: Replica: 01b50763f7a84ce1b25ece17c55f58d7, State: Running, Role: LEADER
I20260928 14:39:56.088501 25390 consensus_queue.cc:237] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } }
I20260928 14:39:56.091563 24915 catalog_manager.cc:5495] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01b50763f7a84ce1b25ece17c55f58d7 (127.6.23.195). New cstate: current_term: 1 leader_uuid: "01b50763f7a84ce1b25ece17c55f58d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01b50763f7a84ce1b25ece17c55f58d7" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 37943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 } health_report { overall_health: UNKNOWN } } }
W20260928 14:39:56.097417 25106 tablet.cc:2369] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:39:56.103829 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.100716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33146 (local address 127.6.23.195:37943)
0928 14:39:56.100837 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:56.100841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:56.100853 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:39:56.100930 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:56.100932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:56.100981 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:56.101064 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:39:56.101268 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.101749 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.102522 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:56.102715 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:56.102811 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:56.103071 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:39:56.103496 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:39:56.103598 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:56.103698 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:39:56.103725 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:39:56.109562 25059 raft_consensus.cc:1261] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 1 FOLLOWER]: Refusing update from remote peer 01b50763f7a84ce1b25ece17c55f58d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:56.109562 25190 raft_consensus.cc:1261] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 1 FOLLOWER]: Refusing update from remote peer 01b50763f7a84ce1b25ece17c55f58d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:56.110239 25390 consensus_queue.cc:1035] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1b4aca2f0eb49cf9a062a66466b84e7" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 41067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:56.110412 25394 consensus_queue.cc:1035] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9c344c2283d4143b5a80ceb2ecc51b6" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:56.118950 25399 mvcc.cc:205] Tried to move back new op lower bound from 7334323798457671680 to 7334323798381068288. Current Snapshot: MvccSnapshot[applied={T|T < 7334323798457671680}]
I20260928 14:39:56.127749 24916 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:56.127919 24916 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:56.130203 24916 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82b2a6e0dd5244cba388658ece3f676a: Sending DeleteTablet for 3 replicas of tablet ef9d14c9724f4f15b85f23504f23c414
I20260928 14:39:56.131026 25301 tablet_service.cc:1514] Processing DeleteTablet for tablet ef9d14c9724f4f15b85f23504f23c414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:56 UTC) from {username='slave'} at 127.0.0.1:33138
I20260928 14:39:56.131031 25170 tablet_service.cc:1514] Processing DeleteTablet for tablet ef9d14c9724f4f15b85f23504f23c414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:56 UTC) from {username='slave'} at 127.0.0.1:53220
I20260928 14:39:56.131031 25039 tablet_service.cc:1514] Processing DeleteTablet for tablet ef9d14c9724f4f15b85f23504f23c414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:56 UTC) from {username='slave'} at 127.0.0.1:56744
I20260928 14:39:56.131541 25408 tablet_replica.cc:330] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7: stopping tablet replica
I20260928 14:39:56.131541 25409 tablet_replica.cc:330] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: stopping tablet replica
I20260928 14:39:56.131851 25408 raft_consensus.cc:2229] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:56.131851 25409 raft_consensus.cc:2229] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:56.132160 25408 raft_consensus.cc:2258] T ef9d14c9724f4f15b85f23504f23c414 P c1b4aca2f0eb49cf9a062a66466b84e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:56.132647 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 24977
I20260928 14:39:56.133493 25409 raft_consensus.cc:2258] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:56.134868 25409 ts_tablet_manager.cc:1905] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:56.137050 25409 ts_tablet_manager.cc:1918] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:56.137117 25409 log.cc:1196] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/ef9d14c9724f4f15b85f23504f23c414
I20260928 14:39:56.137341 25409 ts_tablet_manager.cc:1939] T ef9d14c9724f4f15b85f23504f23c414 P a9c344c2283d4143b5a80ceb2ecc51b6: Deleting consensus metadata
I20260928 14:39:56.138486 24903 catalog_manager.cc:4841] TS a9c344c2283d4143b5a80ceb2ecc51b6 (127.6.23.194:41841): tablet ef9d14c9724f4f15b85f23504f23c414 (table test-table [id=e02f122b695e419c9d80f366c3281d6c]) successfully deleted
I20260928 14:39:56.140026 25407 tablet_replica.cc:330] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: stopping tablet replica
I20260928 14:39:56.140350 25407 raft_consensus.cc:2229] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:39:56.140729 25407 raft_consensus.cc:2258] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:39:56.141436 24903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:41067 (error 108)
I20260928 14:39:56.141652 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 25108
I20260928 14:39:56.142241 25407 ts_tablet_manager.cc:1905] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:56.142800 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:56.142427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:41067 (local address 127.0.0.1:56748)
0928 14:39:56.142628 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:56.142697 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:41067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260928 14:39:56.143075 24903 catalog_manager.cc:4584] TS c1b4aca2f0eb49cf9a062a66466b84e7 (127.6.23.193:41067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef9d14c9724f4f15b85f23504f23c414: Network error: Client connection negotiation failed: client connection to 127.6.23.193:41067: connect: Connection refused (error 111)
I20260928 14:39:56.144649 25407 ts_tablet_manager.cc:1918] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:56.144764 25407 log.cc:1196] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/ef9d14c9724f4f15b85f23504f23c414
I20260928 14:39:56.145049 25407 ts_tablet_manager.cc:1939] T ef9d14c9724f4f15b85f23504f23c414 P 01b50763f7a84ce1b25ece17c55f58d7: Deleting consensus metadata
I20260928 14:39:56.146101 24902 catalog_manager.cc:4841] TS 01b50763f7a84ce1b25ece17c55f58d7 (127.6.23.195:37943): tablet ef9d14c9724f4f15b85f23504f23c414 (table test-table [id=e02f122b695e419c9d80f366c3281d6c]) successfully deleted
I20260928 14:39:56.150506 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 25239
I20260928 14:39:56.155344 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 24886
2026-09-28T14:39:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1022 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:39:56 dist-test-slave-06lh krb5kdc[25413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:39:56 dist-test-slave-06lh krb5kdc[25413](info): set up 2 sockets
Sep 28 14:39:56 dist-test-slave-06lh krb5kdc[25413](info): commencing operation
krb5kdc: starting...
W20260928 14:39:57.191556 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:39:57 dist-test-slave-06lh krb5kdc[25413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:39:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:39:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:39:57.314595 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:34451
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:34451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:57.425455 25426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:57.425733 25426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:57.425776 25426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:57.429903 25426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:39:57.429967 25426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:57.429987 25426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:57.430004 25426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:39:57.430019 25426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:39:57.435436 25426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:34451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:34451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:57.436506 25426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:57.437556 25426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:57.446328 25431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:57.446352 25434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:57.446336 25432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:57.446377 25426 server_base.cc:1034] running on GCE node
I20260928 14:39:57.447160 25426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:57.448319 25426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:57.449488 25426 hybrid_clock.cc:648] HybridClock initialized: now 1790606397449459 us; error 41 us; skew 500 ppm
Sep 28 14:39:57 dist-test-slave-06lh krb5kdc[25413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:57.452462 25426 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:39:57.453416 25426 webserver.cc:460] Webserver started at http://127.6.23.254:42477/ using document root <none> and password file <none>
I20260928 14:39:57.453979 25426 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:57.454041 25426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:57.454304 25426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:57.456449 25426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "fe9eef39edb379101f827bb675edc856"
server_key_iv: "6f58b2f650a2e713b2417e3b3739714d"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.457036 25426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "fe9eef39edb379101f827bb675edc856"
server_key_iv: "6f58b2f650a2e713b2417e3b3739714d"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.460969 25426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:39:57.463457 25441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:57.464704 25426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:39:57.464843 25426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "fe9eef39edb379101f827bb675edc856"
server_key_iv: "6f58b2f650a2e713b2417e3b3739714d"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.464956 25426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:57.482622 25426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:57.485442 25426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:57.485646 25426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:57.492843 25426 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:34451
I20260928 14:39:57.492863 25493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:34451 every 8 connection(s)
I20260928 14:39:57.493765 25426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:39:57.496484 25494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:57.500626 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 25426
I20260928 14:39:57.500762 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:39:57.501042 6239 external_mini_cluster.cc:1428] Setting key d4b4c513c799533a35a8519c5fc7e27c
I20260928 14:39:57.502135 25494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0: Bootstrap starting.
I20260928 14:39:57.504482 25494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:57.505230 25494 log.cc:824] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:57.507131 25494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0: No bootstrap required, opened a new log
Sep 28 14:39:57 dist-test-slave-06lh krb5kdc[25413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:57.510654 25494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34451 } }
I20260928 14:39:57.510979 25494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:57.511073 25494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3783d1d6cf264d2fa868d8b0d05f8ea0, State: Initialized, Role: FOLLOWER
I20260928 14:39:57.511631 25494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34451 } }
I20260928 14:39:57.511792 25494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:39:57.511874 25494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:39:57.511976 25494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:57.512830 25494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34451 } }
I20260928 14:39:57.513163 25494 leader_election.cc:302] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3783d1d6cf264d2fa868d8b0d05f8ea0; no voters:
I20260928 14:39:57.513501 25494 leader_election.cc:288] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:39:57.513568 25499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:57.513765 25499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 3783d1d6cf264d2fa868d8b0d05f8ea0, State: Running, Role: LEADER
I20260928 14:39:57.514066 25499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34451 } }
I20260928 14:39:57.514780 25494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:39:57.515690 25501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34451 } } }
I20260928 14:39:57.515805 25501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:57.516215 25508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:39:57.516196 25500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3783d1d6cf264d2fa868d8b0d05f8ea0. Latest consensus state: current_term: 1 leader_uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3783d1d6cf264d2fa868d8b0d05f8ea0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34451 } } }
I20260928 14:39:57.516251 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:57.502475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.6.23.254:34451)
0928 14:39:57.502921 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:57.502937 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:57.502974 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:39:57.503690 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:57.503709 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:57.504043 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:57.504390 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:57.505209 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.506235 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.507117 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.507691 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.507864 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:57.510322 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:57.510327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:57.510340 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:57.510366 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:57.512422 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:57.512855 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:57.512861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:57.512867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:57.513021 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:57.513313 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:57.513317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:57.513319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:57.513891 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:57.514690 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:57.514812 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:39:57.515128 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260928 14:39:57.516288 25500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:57.519564 25508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:39:57.525511 25508 catalog_manager.cc:1347] Generated new cluster ID: 01f65169b96d4ad6baea32dff31ef356
I20260928 14:39:57.525590 25508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:39:57.545549 25508 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:39:57.546533 25508 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:39:57.554845 25508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0: Generated new TSK 0
I20260928 14:39:57.555619 25508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.23.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
Entry for principal kudu/127.6.23.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab.
I20260928 14:39:57.585871 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34451
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:57.699457 25520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:57.699710 25520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:57.699752 25520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:57.703866 25520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:57.703924 25520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:57.704013 25520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:39:57.709175 25520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.193.keytab
--principal=kudu/127.6.23.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:57.710199 25520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:57.711213 25520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:57.721077 25528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:57.721163 25526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:57.721127 25525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:57.721324 25520 server_base.cc:1034] running on GCE node
I20260928 14:39:57.721911 25520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:57.722482 25520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:57.723637 25520 hybrid_clock.cc:648] HybridClock initialized: now 1790606397723611 us; error 51 us; skew 500 ppm
Sep 28 14:39:57 dist-test-slave-06lh krb5kdc[25413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:57.726413 25520 init.cc:377] Logged in from keytab as kudu/127.6.23.193@KRBTEST.COM (short username kudu)
I20260928 14:39:57.727272 25520 webserver.cc:460] Webserver started at http://127.6.23.193:45875/ using document root <none> and password file <none>
I20260928 14:39:57.727806 25520 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:57.727864 25520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:57.728029 25520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:57.730002 25520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "0354692dc9314eb09c0ffba28e3ec160"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "598236806663684baed259b67da5ef8e"
server_key_iv: "fe1631b82f0eed82a610f80e2f5f8cc1"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.730511 25520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "0354692dc9314eb09c0ffba28e3ec160"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "598236806663684baed259b67da5ef8e"
server_key_iv: "fe1631b82f0eed82a610f80e2f5f8cc1"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.734053 25520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 14:39:57.736402 25535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:57.737648 25520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:39:57.737778 25520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "0354692dc9314eb09c0ffba28e3ec160"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "598236806663684baed259b67da5ef8e"
server_key_iv: "fe1631b82f0eed82a610f80e2f5f8cc1"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.737887 25520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:57.752534 25520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:57.755662 25520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:57.755870 25520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:57.756462 25520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:57.757308 25520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:57.757352 25520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:57.757419 25520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:57.757450 25520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:57.766192 25520 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:32915
I20260928 14:39:57.766223 25648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:32915 every 8 connection(s)
I20260928 14:39:57.767072 25520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:39:57.772317 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 25520
I20260928 14:39:57.772428 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:39:57.772745 6239 external_mini_cluster.cc:1428] Setting key 73a81caa4c49426184f8739c578fc5a4
Sep 28 14:39:57 dist-test-slave-06lh krb5kdc[25413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.193@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:57.782156 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:57.768980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:59507 (local address 127.6.23.254:34451)
0928 14:39:57.769131 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:57.769136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:57.770308 (+ 1172us) server_negotiation.cc:407] Connection header received
0928 14:39:57.771259 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:57.771263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:57.771325 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:57.771428 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:57.772712 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.773331 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.774802 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.775250 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.775368 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:57.778126 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:57.778132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:57.778138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:57.778174 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:57.779883 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:57.780543 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:57.780545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:57.780547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:57.780634 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:57.781028 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:57.781030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:57.781031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:57.781203 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:57.781277 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:57.781938 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 14:39:57.782037 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.6.23.194@KRBTEST.COM; defaulting to no policy
I20260928 14:39:57.783407 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:57.769455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34451 (local address 127.6.23.193:59507)
0928 14:39:57.770129 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:57.770178 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:57.771019 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:57.771609 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:57.771615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:57.771987 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:57.772561 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:57.772578 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.773480 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.773484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:57.774668 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:57.774680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.775554 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.775560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:57.775813 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:57.776501 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:57.776517 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:57.777960 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:57.780008 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:57.780018 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:57.780036 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:57.780431 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:57.780741 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:57.780745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:57.780747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:57.780931 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:57.781328 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:57.781337 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:57.781677 (+ 340us) client_negotiation.cc:769] Sending connection context
0928 14:39:57.781907 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 14:39:57.782293 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":114,"threads_started":1}
I20260928 14:39:57.784766 25649 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34451
I20260928 14:39:57.785017 25649 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.194@KRBTEST.COM" created.
I20260928 14:39:57.785665 25649 heartbeater.cc:511] Master 127.6.23.254:34451 requested a full tablet report, sending...
I20260928 14:39:57.787525 25458 ts_manager.cc:194] Registered new tserver with Master: 0354692dc9314eb09c0ffba28e3ec160 (127.6.23.193:32915)
I20260928 14:39:57.788687 25458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.193@KRBTEST.COM'} at 127.6.23.193:59507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
Entry for principal kudu/127.6.23.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab.
I20260928 14:39:57.802599 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34451
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:57.914229 25654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:57.914546 25654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:57.914623 25654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:57.919447 25654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:57.919540 25654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:57.919641 25654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:39:57.924996 25654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.194.keytab
--principal=kudu/127.6.23.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:57.926090 25654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:57.927130 25654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:39:57.936877 25654 server_base.cc:1034] running on GCE node
W20260928 14:39:57.936854 25659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:57.936968 25660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:57.936848 25662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:57.937459 25654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:57.938181 25654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:57.939311 25654 hybrid_clock.cc:648] HybridClock initialized: now 1790606397939290 us; error 31 us; skew 500 ppm
Sep 28 14:39:57 dist-test-slave-06lh krb5kdc[25413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:57.942239 25654 init.cc:377] Logged in from keytab as kudu/127.6.23.194@KRBTEST.COM (short username kudu)
I20260928 14:39:57.943094 25654 webserver.cc:460] Webserver started at http://127.6.23.194:40551/ using document root <none> and password file <none>
I20260928 14:39:57.943702 25654 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:57.943800 25654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:57.944024 25654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:57.946038 25654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "599809bd24dc4942b9ad2813d6f713f9"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "edfabdd557e88295ce5ed4f69c07f9b5"
server_key_iv: "56f8b7c0485157be854fad40bf62d180"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.946587 25654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "599809bd24dc4942b9ad2813d6f713f9"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "edfabdd557e88295ce5ed4f69c07f9b5"
server_key_iv: "56f8b7c0485157be854fad40bf62d180"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.950322 25654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:39:57.952754 25669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:57.953925 25654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:39:57.954059 25654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "599809bd24dc4942b9ad2813d6f713f9"
format_stamp: "Formatted at 2026-09-28 14:39:57 on dist-test-slave-06lh"
server_key: "edfabdd557e88295ce5ed4f69c07f9b5"
server_key_iv: "56f8b7c0485157be854fad40bf62d180"
server_key_version: "encryptionkey@0"
I20260928 14:39:57.954170 25654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:57.974393 25654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:57.977840 25654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:57.978057 25654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:57.978734 25654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:57.979611 25654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:57.979682 25654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:57.979753 25654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:57.979801 25654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:57.988778 25654 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:33431
I20260928 14:39:57.988797 25782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:33431 every 8 connection(s)
I20260928 14:39:57.989699 25654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:39:57.998931 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 25654
Sep 28 14:39:57 dist-test-slave-06lh krb5kdc[25413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606397, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.194@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:57.999091 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:39:57.999475 6239 external_mini_cluster.cc:1428] Setting key c7d097ff7dc2a8bfe474fedcb62dd39f
I20260928 14:39:58.003603 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:57.991691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:47977 (local address 127.6.23.254:34451)
0928 14:39:57.991840 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:57.991845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:57.992769 (+ 924us) server_negotiation.cc:407] Connection header received
0928 14:39:57.993643 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:57.993646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:57.993694 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:57.993770 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:57.995002 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.995635 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.996818 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.997027 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.997106 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:57.999738 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:57.999741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:57.999743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:57.999767 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:58.001460 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:58.001960 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:58.001966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:58.001970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:58.002065 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:58.002379 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:58.002385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:58.002390 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:58.002639 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:58.002749 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.003391 (+ 642us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.003489 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:39:58.004858 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:57.992128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34451 (local address 127.6.23.194:47977)
0928 14:39:57.992600 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:57.992649 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:57.993444 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:57.993930 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:57.993937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:57.994336 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:57.994845 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:57.994859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.995799 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.995802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:57.996703 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:57.996714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:57.997160 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:57.997163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:57.997334 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:57.997969 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:57.997991 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:57.999569 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:58.001584 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:58.001589 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:58.001601 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:58.001863 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:58.002154 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:58.002156 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:58.002158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:58.002288 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:58.002766 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:58.002777 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:58.003109 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 14:39:58.003348 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 14:39:58.003722 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260928 14:39:58.006222 25783 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34451
I20260928 14:39:58.006484 25783 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:58.006984 25783 heartbeater.cc:511] Master 127.6.23.254:34451 requested a full tablet report, sending...
I20260928 14:39:58.008167 25458 ts_manager.cc:194] Registered new tserver with Master: 599809bd24dc4942b9ad2813d6f713f9 (127.6.23.194:33431)
I20260928 14:39:58.008991 25458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.194@KRBTEST.COM'} at 127.6.23.194:47977
WARNING: no policy specified for kudu/127.6.23.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
Entry for principal kudu/127.6.23.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab.
I20260928 14:39:58.028534 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34451
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:39:58.139336 25788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:58.139647 25788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:58.139730 25788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:58.143913 25788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:58.143991 25788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:58.144079 25788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:39:58.149384 25788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.195.keytab
--principal=kudu/127.6.23.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:58.150507 25788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:58.151576 25788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:58.161321 25793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:58.161324 25796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:58.161335 25794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:58.161389 25788 server_base.cc:1034] running on GCE node
I20260928 14:39:58.161916 25788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:58.162570 25788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:58.163757 25788 hybrid_clock.cc:648] HybridClock initialized: now 1790606398163734 us; error 33 us; skew 500 ppm
Sep 28 14:39:58 dist-test-slave-06lh krb5kdc[25413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:39:58.166404 25788 init.cc:377] Logged in from keytab as kudu/127.6.23.195@KRBTEST.COM (short username kudu)
I20260928 14:39:58.167343 25788 webserver.cc:460] Webserver started at http://127.6.23.195:46645/ using document root <none> and password file <none>
I20260928 14:39:58.167894 25788 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:58.167980 25788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:58.168267 25788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:58.170347 25788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "e3e000892db54d9c8cbc8c1927fa8551"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "67ea1ab4af09872a7feec758f1fefaa5"
server_key_iv: "e6839baf28ea425999f772853b9b7d69"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.170905 25788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "e3e000892db54d9c8cbc8c1927fa8551"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "67ea1ab4af09872a7feec758f1fefaa5"
server_key_iv: "e6839baf28ea425999f772853b9b7d69"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.174623 25788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:39:58.177106 25803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:58.178371 25788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:39:58.178508 25788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "e3e000892db54d9c8cbc8c1927fa8551"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "67ea1ab4af09872a7feec758f1fefaa5"
server_key_iv: "e6839baf28ea425999f772853b9b7d69"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.178617 25788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:58.197510 25788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:58.200325 25788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:58.200523 25788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:58.201098 25788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:58.201910 25788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:58.201977 25788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:58.202049 25788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:58.202090 25788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:58.211340 25788 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:42579
I20260928 14:39:58.211359 25916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:42579 every 8 connection(s)
I20260928 14:39:58.212257 25788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:39:58.214829 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 25788
I20260928 14:39:58.214967 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:39:58.215351 6239 external_mini_cluster.cc:1428] Setting key 4dc0309e8523ad0055c4ed72dbd4d08f
Sep 28 14:39:58 dist-test-slave-06lh krb5kdc[25413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606398, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.195@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:39:58.225533 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.213991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:34621 (local address 127.6.23.254:34451)
0928 14:39:58.214120 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.214131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.215070 (+ 939us) server_negotiation.cc:407] Connection header received
0928 14:39:58.216255 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.216258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.216316 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.216393 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:58.217581 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.218119 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.219274 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.219497 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.219610 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.222032 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:58.222036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:58.222039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:58.222068 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:58.223531 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:58.224034 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:58.224038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:58.224041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:58.224124 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:58.224463 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:58.224468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:58.224470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:58.224652 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:58.224755 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.225241 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.225368 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:39:58.226313 25919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.214375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34451 (local address 127.6.23.195:34621)
0928 14:39:58.214895 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:58.214962 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:58.216050 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:58.216609 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:58.216617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:58.216961 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:58.217443 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.217456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.218245 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.218248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.219123 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.219132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.219643 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.219646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.219823 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.220482 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:39:58.220498 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:58.221875 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:58.223647 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:58.223652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:58.223667 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:58.223928 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:58.224216 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:39:58.224220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:39:58.224222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:39:58.224367 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:39:58.224763 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:58.224771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:58.225011 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 14:39:58.225199 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 14:39:58.225522 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260928 14:39:58.227523 25917 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34451
I20260928 14:39:58.227757 25917 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:58.228233 25917 heartbeater.cc:511] Master 127.6.23.254:34451 requested a full tablet report, sending...
I20260928 14:39:58.229324 25458 ts_manager.cc:194] Registered new tserver with Master: e3e000892db54d9c8cbc8c1927fa8551 (127.6.23.195:42579)
I20260928 14:39:58.229912 25458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.23.195@KRBTEST.COM'} at 127.6.23.195:34621
I20260928 14:39:58.241108 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:39:58.250661 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.243307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.6.23.254:34451)
0928 14:39:58.243448 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.243454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.243804 (+ 350us) server_negotiation.cc:407] Connection header received
0928 14:39:58.243901 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.243903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.243943 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.244003 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:58.244686 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.245195 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.246040 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.246250 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.246354 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.247781 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:58.247784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:58.247786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:39:58.247809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:58.248998 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:58.249493 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:58.249495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:58.249497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:58.249563 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:39:58.249829 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:39:58.249831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:39:58.249832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:39:58.250034 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:58.250136 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.250339 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.250476 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:39:58.253278 25458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:39:58.256263 25458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:39:58.271191 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.265720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.6.23.194:33431)
0928 14:39:58.266077 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.266091 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.266118 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:39:58.266200 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.266204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.266362 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.266516 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:58.266801 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.268131 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.269701 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.270541 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.270651 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.270811 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.270934 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.271032 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":120,"threads_started":1}
I20260928 14:39:58.271183 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.264699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:33431 (local address 127.0.0.1:51562)
0928 14:39:58.265038 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:58.265078 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:58.265215 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:58.266507 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:58.266513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:58.266535 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:58.266673 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.266680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.268527 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.268532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.269573 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.269582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.270663 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.270666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.270753 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.270782 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:39:58.270906 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:39:58.271030 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":34,"thread_start_us":83,"threads_started":1}
I20260928 14:39:58.271559 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.265573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.6.23.195:42579)
0928 14:39:58.266169 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.266183 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.266207 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:39:58.266280 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.266285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.266479 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.266630 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:58.267089 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.268238 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.269968 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.270745 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.270853 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.270981 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.271311 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.271414 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":105,"threads_started":1}
I20260928 14:39:58.271201 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.265422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:42579 (local address 127.0.0.1:60780)
0928 14:39:58.265951 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:58.265966 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:58.266052 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:58.266787 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:58.266791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:58.266816 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:58.266963 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.266969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.268377 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.268381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.269842 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.269853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.270864 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.270870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.270964 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.270984 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:39:58.271047 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:39:58.271100 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":125,"spinlock_wait_cycles":384,"thread_start_us":91,"threads_started":1}
I20260928 14:39:58.273088 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.264776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:32915 (local address 127.0.0.1:41910)
0928 14:39:58.265827 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:58.265845 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:58.265936 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:58.266787 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:58.266791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:58.266821 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:58.266963 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.266969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.268180 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.268186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.269842 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.269853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.272660 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.272664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.272774 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.272793 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:39:58.272884 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:39:58.272957 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":246,"spinlock_wait_cycles":15360,"thread_start_us":88,"threads_started":1}
I20260928 14:39:58.273152 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.265493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.6.23.193:32915)
0928 14:39:58.266169 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.266188 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.266207 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:39:58.266284 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.266288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.266479 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.266630 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:58.267121 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.268032 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.271039 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.272363 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.272466 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.272593 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.272917 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.273005 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":67,"threads_started":1}
I20260928 14:39:58.274845 25717 tablet_service.cc:1467] Processing CreateTablet for tablet a6d463954b2345f5b8f69d87b2ecd5ad (DEFAULT_TABLE table=test-table [id=7d144f2d67a243409ca7c15ba2291390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:58.274838 25851 tablet_service.cc:1467] Processing CreateTablet for tablet a6d463954b2345f5b8f69d87b2ecd5ad (DEFAULT_TABLE table=test-table [id=7d144f2d67a243409ca7c15ba2291390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:58.276048 25717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d463954b2345f5b8f69d87b2ecd5ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:58.276114 25851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d463954b2345f5b8f69d87b2ecd5ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:58.281770 25583 tablet_service.cc:1467] Processing CreateTablet for tablet a6d463954b2345f5b8f69d87b2ecd5ad (DEFAULT_TABLE table=test-table [id=7d144f2d67a243409ca7c15ba2291390]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:58.282778 25583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6d463954b2345f5b8f69d87b2ecd5ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:58.283602 25934 tablet_bootstrap.cc:492] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: Bootstrap starting.
I20260928 14:39:58.284009 25935 tablet_bootstrap.cc:492] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: Bootstrap starting.
I20260928 14:39:58.286798 25935 tablet_bootstrap.cc:654] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:58.287731 25935 log.cc:824] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:58.289103 25934 tablet_bootstrap.cc:654] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:58.289836 25935 tablet_bootstrap.cc:492] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: No bootstrap required, opened a new log
I20260928 14:39:58.290030 25935 ts_tablet_manager.cc:1397] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260928 14:39:58.290201 25934 log.cc:824] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:58.290225 25937 tablet_bootstrap.cc:492] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160: Bootstrap starting.
I20260928 14:39:58.292172 25937 tablet_bootstrap.cc:654] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:58.292961 25934 tablet_bootstrap.cc:492] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: No bootstrap required, opened a new log
I20260928 14:39:58.293000 25937 log.cc:824] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:58.293155 25934 ts_tablet_manager.cc:1397] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 14:39:58.293846 25935 raft_consensus.cc:348] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.294092 25935 raft_consensus.cc:374] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:58.294144 25935 raft_consensus.cc:729] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599809bd24dc4942b9ad2813d6f713f9, State: Initialized, Role: FOLLOWER
I20260928 14:39:58.294580 25935 consensus_queue.cc:260] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.294906 25937 tablet_bootstrap.cc:492] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160: No bootstrap required, opened a new log
I20260928 14:39:58.295090 25937 ts_tablet_manager.cc:1397] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:39:58.295217 25783 heartbeater.cc:503] Master 127.6.23.254:34451 was elected leader, sending a full tablet report...
I20260928 14:39:58.295555 25935 ts_tablet_manager.cc:1428] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 14:39:58.297009 25934 raft_consensus.cc:348] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.297272 25934 raft_consensus.cc:374] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:58.297326 25934 raft_consensus.cc:729] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3e000892db54d9c8cbc8c1927fa8551, State: Initialized, Role: FOLLOWER
I20260928 14:39:58.297771 25934 consensus_queue.cc:260] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.298700 25934 ts_tablet_manager.cc:1428] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 14:39:58.298822 25917 heartbeater.cc:503] Master 127.6.23.254:34451 was elected leader, sending a full tablet report...
I20260928 14:39:58.299427 25937 raft_consensus.cc:348] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.299646 25937 raft_consensus.cc:374] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:58.299705 25937 raft_consensus.cc:729] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0354692dc9314eb09c0ffba28e3ec160, State: Initialized, Role: FOLLOWER
I20260928 14:39:58.300150 25937 consensus_queue.cc:260] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.301118 25937 ts_tablet_manager.cc:1428] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 14:39:58.301452 25649 heartbeater.cc:503] Master 127.6.23.254:34451 was elected leader, sending a full tablet report...
I20260928 14:39:58.341447 25942 raft_consensus.cc:482] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:58.341674 25942 raft_consensus.cc:504] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.342927 25942 leader_election.cc:288] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 599809bd24dc4942b9ad2813d6f713f9 (127.6.23.194:33431), e3e000892db54d9c8cbc8c1927fa8551 (127.6.23.195:42579)
I20260928 14:39:58.346818 25933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.343190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:37423 (local address 127.6.23.194:33431)
0928 14:39:58.343378 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.343383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.343552 (+ 169us) server_negotiation.cc:407] Connection header received
0928 14:39:58.343722 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.343727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.343793 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.343885 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:58.344265 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.344771 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.345843 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.346257 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.346391 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.346445 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.346619 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.346688 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 14:39:58.346839 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.343073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:33431 (local address 127.6.23.193:37423)
0928 14:39:58.343461 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:58.343474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:58.343597 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:58.344015 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:58.344018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:58.344037 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:58.344157 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.344161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.344901 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.344906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.345714 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.345723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.346385 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.346388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.346484 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.346514 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:39:58.346604 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 14:39:58.346675 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"spinlock_wait_cycles":3328,"thread_start_us":234,"threads_started":1}
I20260928 14:39:58.347085 25932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.343792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:37899 (local address 127.6.23.195:42579)
0928 14:39:58.343945 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.343949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.344032 (+ 83us) server_negotiation.cc:407] Connection header received
0928 14:39:58.344189 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.344192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.344247 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.344330 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:58.344598 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.345085 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.346117 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.346522 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.346649 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.346696 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.346875 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.346968 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:39:58.347515 25944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.343626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:42579 (local address 127.6.23.193:37899)
0928 14:39:58.343940 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:58.343951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:58.344074 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:58.344339 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:58.344341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:58.344356 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:58.344491 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.344496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.345191 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.345198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.345993 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.346001 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.346640 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.346644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.346745 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.346765 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:39:58.346860 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:39:58.347383 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
I20260928 14:39:58.347617 25737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d463954b2345f5b8f69d87b2ecd5ad" candidate_uuid: "0354692dc9314eb09c0ffba28e3ec160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599809bd24dc4942b9ad2813d6f713f9" is_pre_election: true
I20260928 14:39:58.347982 25737 raft_consensus.cc:2454] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0354692dc9314eb09c0ffba28e3ec160 in term 0.
I20260928 14:39:58.348117 25871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d463954b2345f5b8f69d87b2ecd5ad" candidate_uuid: "0354692dc9314eb09c0ffba28e3ec160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e000892db54d9c8cbc8c1927fa8551" is_pre_election: true
I20260928 14:39:58.348423 25871 raft_consensus.cc:2454] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0354692dc9314eb09c0ffba28e3ec160 in term 0.
I20260928 14:39:58.348534 25539 leader_election.cc:302] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0354692dc9314eb09c0ffba28e3ec160, 599809bd24dc4942b9ad2813d6f713f9; no voters:
I20260928 14:39:58.348829 25942 raft_consensus.cc:2781] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:39:58.348924 25942 raft_consensus.cc:482] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:39:58.348981 25942 raft_consensus.cc:3037] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:58.350013 25942 raft_consensus.cc:504] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.350346 25942 leader_election.cc:288] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [CANDIDATE]: Term 1 election: Requested vote from peers 599809bd24dc4942b9ad2813d6f713f9 (127.6.23.194:33431), e3e000892db54d9c8cbc8c1927fa8551 (127.6.23.195:42579)
I20260928 14:39:58.350688 25737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d463954b2345f5b8f69d87b2ecd5ad" candidate_uuid: "0354692dc9314eb09c0ffba28e3ec160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "599809bd24dc4942b9ad2813d6f713f9"
I20260928 14:39:58.350824 25737 raft_consensus.cc:3037] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:58.350878 25871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6d463954b2345f5b8f69d87b2ecd5ad" candidate_uuid: "0354692dc9314eb09c0ffba28e3ec160" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3e000892db54d9c8cbc8c1927fa8551"
I20260928 14:39:58.351032 25871 raft_consensus.cc:3037] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:58.351970 25737 raft_consensus.cc:2454] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0354692dc9314eb09c0ffba28e3ec160 in term 1.
I20260928 14:39:58.352164 25871 raft_consensus.cc:2454] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0354692dc9314eb09c0ffba28e3ec160 in term 1.
I20260928 14:39:58.352308 25539 leader_election.cc:302] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0354692dc9314eb09c0ffba28e3ec160, 599809bd24dc4942b9ad2813d6f713f9; no voters:
I20260928 14:39:58.352478 25942 raft_consensus.cc:2781] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:58.352718 25942 raft_consensus.cc:686] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [term 1 LEADER]: Becoming Leader. State: Replica: 0354692dc9314eb09c0ffba28e3ec160, State: Running, Role: LEADER
I20260928 14:39:58.353080 25942 consensus_queue.cc:237] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } }
I20260928 14:39:58.356431 25457 catalog_manager.cc:5495] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0354692dc9314eb09c0ffba28e3ec160 (127.6.23.193). New cstate: current_term: 1 leader_uuid: "0354692dc9314eb09c0ffba28e3ec160" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0354692dc9314eb09c0ffba28e3ec160" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 32915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 } health_report { overall_health: UNKNOWN } } }
I20260928 14:39:58.366803 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.363479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.6.23.193:32915)
0928 14:39:58.363599 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.363603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.363615 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:39:58.363683 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.363685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.363741 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.363837 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:39:58.364147 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.364670 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.365549 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.365759 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.365845 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.366093 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:39:58.366502 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:39:58.366590 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.366648 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.366697 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:39:58.372710 25871 raft_consensus.cc:1261] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 1 FOLLOWER]: Refusing update from remote peer 0354692dc9314eb09c0ffba28e3ec160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:58.372710 25737 raft_consensus.cc:1261] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 1 FOLLOWER]: Refusing update from remote peer 0354692dc9314eb09c0ffba28e3ec160: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:58.373447 25942 consensus_queue.cc:1035] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "599809bd24dc4942b9ad2813d6f713f9" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 33431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:58.373690 25945 consensus_queue.cc:1035] T a6d463954b2345f5b8f69d87b2ecd5ad P 0354692dc9314eb09c0ffba28e3ec160 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3e000892db54d9c8cbc8c1927fa8551" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:58.382045 25948 mvcc.cc:205] Tried to move back new op lower bound from 7334323807727411200 to 7334323807656837120. Current Snapshot: MvccSnapshot[applied={T|T < 7334323807727411200}]
I20260928 14:39:58.382227 25949 mvcc.cc:205] Tried to move back new op lower bound from 7334323807727411200 to 7334323807656837120. Current Snapshot: MvccSnapshot[applied={T|T < 7334323807727411200}]
I20260928 14:39:58.382510 25950 mvcc.cc:205] Tried to move back new op lower bound from 7334323807727411200 to 7334323807656837120. Current Snapshot: MvccSnapshot[applied={T|T < 7334323807727411200}]
I20260928 14:39:58.391448 25457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:58.391654 25457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:58.394573 25457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3783d1d6cf264d2fa868d8b0d05f8ea0: Sending DeleteTablet for 3 replicas of tablet a6d463954b2345f5b8f69d87b2ecd5ad
I20260928 14:39:58.395686 25851 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d463954b2345f5b8f69d87b2ecd5ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:58 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:60780
I20260928 14:39:58.395983 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 25520
I20260928 14:39:58.396100 25583 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d463954b2345f5b8f69d87b2ecd5ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:58 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:41910
I20260928 14:39:58.396665 25717 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d463954b2345f5b8f69d87b2ecd5ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:58 UTC) from {username='kudu', principal='kudu/127.6.23.254@KRBTEST.COM'} at 127.0.0.1:51562
I20260928 14:39:58.397161 25959 tablet_replica.cc:330] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: stopping tablet replica
I20260928 14:39:58.397540 25959 raft_consensus.cc:2229] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:58.397886 25958 tablet_replica.cc:330] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: stopping tablet replica
I20260928 14:39:58.398185 25958 raft_consensus.cc:2229] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:58.398432 25959 raft_consensus.cc:2258] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:58.398455 25958 raft_consensus.cc:2258] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:58.400447 25958 ts_tablet_manager.cc:1905] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:58.400447 25959 ts_tablet_manager.cc:1905] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:58.403069 25958 ts_tablet_manager.cc:1918] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:58.403069 25959 ts_tablet_manager.cc:1918] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:58.403172 25958 log.cc:1196] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/a6d463954b2345f5b8f69d87b2ecd5ad
I20260928 14:39:58.403321 25959 log.cc:1196] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/a6d463954b2345f5b8f69d87b2ecd5ad
I20260928 14:39:58.403636 25958 ts_tablet_manager.cc:1939] T a6d463954b2345f5b8f69d87b2ecd5ad P e3e000892db54d9c8cbc8c1927fa8551: Deleting consensus metadata
I20260928 14:39:58.403781 25959 ts_tablet_manager.cc:1939] T a6d463954b2345f5b8f69d87b2ecd5ad P 599809bd24dc4942b9ad2813d6f713f9: Deleting consensus metadata
W20260928 14:39:58.403817 25444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:32915 (error 108)
I20260928 14:39:58.404910 25445 catalog_manager.cc:4841] TS e3e000892db54d9c8cbc8c1927fa8551 (127.6.23.195:42579): tablet a6d463954b2345f5b8f69d87b2ecd5ad (table test-table [id=7d144f2d67a243409ca7c15ba2291390]) successfully deleted
I20260928 14:39:58.405121 25445 catalog_manager.cc:4841] TS 599809bd24dc4942b9ad2813d6f713f9 (127.6.23.194:33431): tablet a6d463954b2345f5b8f69d87b2ecd5ad (table test-table [id=7d144f2d67a243409ca7c15ba2291390]) successfully deleted
W20260928 14:39:58.405342 25929 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:39:58.404723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:32915 (local address 127.0.0.1:36036)
0928 14:39:58.404938 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:58.404950 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:58.405038 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:58.405246 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:32915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":132}
W20260928 14:39:58.405488 25444 catalog_manager.cc:4584] TS 0354692dc9314eb09c0ffba28e3ec160 (127.6.23.193:32915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6d463954b2345f5b8f69d87b2ecd5ad: Network error: Client connection negotiation failed: client connection to 127.6.23.193:32915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:39:58.405550 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 25654
I20260928 14:39:58.411767 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 25788
I20260928 14:39:58.417310 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 25426
2026-09-28T14:39:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2262 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T14:39:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:39:58Z Disabled control of system clock
I20260928 14:39:58.451195 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:34813
--webserver_interface=127.6.23.254
--webserver_port=0
--builtin_ntp_servers=127.6.23.212:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:34813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:39:58.563966 25964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:58.564235 25964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:58.564271 25964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:58.568451 25964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:39:58.568507 25964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:58.568522 25964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:58.568564 25964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:39:58.568600 25964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:39:58.573299 25964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:34813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:34813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:58.574381 25964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:58.575811 25964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:58.584690 25972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:58.584738 25970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:58.584715 25964 server_base.cc:1034] running on GCE node
W20260928 14:39:58.584748 25969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:58.585532 25964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:58.586638 25964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:58.587827 25964 hybrid_clock.cc:648] HybridClock initialized: now 1790606398587813 us; error 29 us; skew 500 ppm
I20260928 14:39:58.589651 25964 webserver.cc:460] Webserver started at http://127.6.23.254:34691/ using document root <none> and password file <none>
I20260928 14:39:58.590241 25964 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:58.590341 25964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:58.590574 25964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:58.592660 25964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "452d8ae4e6b24087acdbe076ed3b6bc0"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "e3331d57cf812ca2631c80e9f9bf9c38"
server_key_iv: "85c56b13d3416717eda1e1975c140936"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.593235 25964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "452d8ae4e6b24087acdbe076ed3b6bc0"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "e3331d57cf812ca2631c80e9f9bf9c38"
server_key_iv: "85c56b13d3416717eda1e1975c140936"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.596925 25964 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 14:39:58.599335 25978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:58.600508 25964 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:58.600649 25964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "452d8ae4e6b24087acdbe076ed3b6bc0"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "e3331d57cf812ca2631c80e9f9bf9c38"
server_key_iv: "85c56b13d3416717eda1e1975c140936"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.600760 25964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:58.613866 25964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:58.614537 25964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:58.614737 25964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:58.621557 25964 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:34813
I20260928 14:39:58.621573 26030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:34813 every 8 connection(s)
I20260928 14:39:58.622478 25964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:39:58.625622 26031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:58.627120 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 25964
I20260928 14:39:58.627297 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:39:58.627616 6239 external_mini_cluster.cc:1428] Setting key c919377de5ab06884936aac3d395b612
I20260928 14:39:58.631915 26031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0: Bootstrap starting.
I20260928 14:39:58.634054 26031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:58.634781 26031 log.cc:824] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:58.636595 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.629016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40836 (local address 127.6.23.254:34813)
0928 14:39:58.629652 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.629666 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.629700 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:39:58.630358 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.630374 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.630744 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.631072 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:58.631513 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.632499 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.633367 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.633838 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.634013 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.634862 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:58.634867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:58.634879 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:58.635357 (+ 478us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:58.635413 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:58.635433 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:58.635538 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.635654 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.635912 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
I20260928 14:39:58.636799 26031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0: No bootstrap required, opened a new log
I20260928 14:39:58.640368 26031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452d8ae4e6b24087acdbe076ed3b6bc0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34813 } }
I20260928 14:39:58.640600 26031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:58.640678 26031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 452d8ae4e6b24087acdbe076ed3b6bc0, State: Initialized, Role: FOLLOWER
I20260928 14:39:58.641146 26031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452d8ae4e6b24087acdbe076ed3b6bc0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34813 } }
I20260928 14:39:58.641300 26031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:39:58.641387 26031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:39:58.641484 26031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:58.642485 26031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452d8ae4e6b24087acdbe076ed3b6bc0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34813 } }
I20260928 14:39:58.642824 26031 leader_election.cc:302] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 452d8ae4e6b24087acdbe076ed3b6bc0; no voters:
I20260928 14:39:58.643110 26031 leader_election.cc:288] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:39:58.643222 26036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:58.643600 26036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 452d8ae4e6b24087acdbe076ed3b6bc0, State: Running, Role: LEADER
I20260928 14:39:58.643936 26036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452d8ae4e6b24087acdbe076ed3b6bc0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34813 } }
I20260928 14:39:58.644574 26031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:39:58.645725 26037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "452d8ae4e6b24087acdbe076ed3b6bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452d8ae4e6b24087acdbe076ed3b6bc0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34813 } } }
I20260928 14:39:58.645844 26037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:58.646040 26038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 452d8ae4e6b24087acdbe076ed3b6bc0. Latest consensus state: current_term: 1 leader_uuid: "452d8ae4e6b24087acdbe076ed3b6bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "452d8ae4e6b24087acdbe076ed3b6bc0" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 34813 } } }
I20260928 14:39:58.646142 26038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:39:58.646858 26045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:39:58.649472 26045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:39:58.654709 26045 catalog_manager.cc:1347] Generated new cluster ID: dfa7ee4ac279449cbfc82aa8da6cc614
I20260928 14:39:58.654783 26045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:39:58.673539 26045 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:39:58.674561 26045 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:39:58.679080 26045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0: Generated new TSK 0
I20260928 14:39:58.680007 26045 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:39:58.691324 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.193:0
--local_ip_for_outbound_sockets=127.6.23.193
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34813
--builtin_ntp_servers=127.6.23.212:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:39:58.803479 26055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:58.803773 26055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:58.803843 26055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:58.807976 26055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:58.808027 26055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:58.808082 26055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.193
I20260928 14:39:58.812717 26055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.23.193
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.193
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:58.813766 26055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:58.815131 26055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:58.824743 26060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:58.824756 26061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:58.824750 26063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:58.824965 26055 server_base.cc:1034] running on GCE node
I20260928 14:39:58.825424 26055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:58.826042 26055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:58.827270 26055 hybrid_clock.cc:648] HybridClock initialized: now 1790606398827226 us; error 62 us; skew 500 ppm
I20260928 14:39:58.829191 26055 webserver.cc:460] Webserver started at http://127.6.23.193:46285/ using document root <none> and password file <none>
I20260928 14:39:58.829775 26055 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:58.829871 26055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:58.830104 26055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:58.832228 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/data/instance:
uuid: "f03578579ee54495aa10765b3576e2d8"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "4cb94e12fa986487653f4bd1f374a5ca"
server_key_iv: "86e72c01929a7d2fa9bd3abef8fe372b"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.832773 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance:
uuid: "f03578579ee54495aa10765b3576e2d8"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "4cb94e12fa986487653f4bd1f374a5ca"
server_key_iv: "86e72c01929a7d2fa9bd3abef8fe372b"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.836416 26055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:39:58.838814 26069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:58.840065 26055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:39:58.840196 26055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/wal
uuid: "f03578579ee54495aa10765b3576e2d8"
format_stamp: "Formatted at 2026-09-28 14:39:58 on dist-test-slave-06lh"
server_key: "4cb94e12fa986487653f4bd1f374a5ca"
server_key_iv: "86e72c01929a7d2fa9bd3abef8fe372b"
server_key_version: "encryptionkey@0"
I20260928 14:39:58.840306 26055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:58.855875 26055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:58.856570 26055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:58.856771 26055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:58.857383 26055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:58.858395 26055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:58.858469 26055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:58.858541 26055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:58.858585 26055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:58.867720 26055 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.193:40407
I20260928 14:39:58.867741 26182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.193:40407 every 8 connection(s)
I20260928 14:39:58.868669 26055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/data/info.pb
I20260928 14:39:58.877707 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.870645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.193:46829 (local address 127.6.23.254:34813)
0928 14:39:58.870796 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:58.870800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:58.871936 (+ 1136us) server_negotiation.cc:407] Connection header received
0928 14:39:58.873262 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:58.873264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:58.873317 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:58.873401 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:58.874289 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.874834 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.875995 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.876210 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.876288 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.877296 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:58.877299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:58.877301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:58.877312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:58.877344 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:58.877352 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:58.877410 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:58.877544 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:39:58.877595 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:39:58.878206 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 26055
I20260928 14:39:58.878350 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-0/wal/instance
I20260928 14:39:58.878624 26185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:58.871073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34813 (local address 127.6.23.193:46829)
0928 14:39:58.871772 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:58.871814 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:58.873072 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:58.873571 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:58.873577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:58.873661 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:58.874169 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.874182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.874944 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.874947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.875893 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:58.875902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:58.876354 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:58.876356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:58.876497 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:58.877058 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:58.877059 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:58.877099 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:58.877160 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:58.877424 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:58.877430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:58.877434 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:39:58.877523 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:39:58.877732 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
I20260928 14:39:58.878652 6239 external_mini_cluster.cc:1428] Setting key 66936438d0b24ead4f1561fbd95e8fe0
I20260928 14:39:58.880048 26183 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34813
I20260928 14:39:58.880307 26183 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:58.880815 26183 heartbeater.cc:511] Master 127.6.23.254:34813 requested a full tablet report, sending...
I20260928 14:39:58.880868 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.194:0
--local_ip_for_outbound_sockets=127.6.23.194
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34813
--builtin_ntp_servers=127.6.23.212:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:39:58.882480 25995 ts_manager.cc:194] Registered new tserver with Master: f03578579ee54495aa10765b3576e2d8 (127.6.23.193:40407)
I20260928 14:39:58.884160 25995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.193:46829
W20260928 14:39:58.992827 26186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:58.993093 26186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:58.993134 26186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:58.997079 26186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:58.997128 26186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:58.997186 26186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.194
I20260928 14:39:59.002393 26186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.23.194
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.194
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:59.003525 26186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:59.004914 26186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:59.014711 26194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:59.014737 26186 server_base.cc:1034] running on GCE node
W20260928 14:39:59.014681 26191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:59.014706 26192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:59.015421 26186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:59.016019 26186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:59.017186 26186 hybrid_clock.cc:648] HybridClock initialized: now 1790606399017141 us; error 53 us; skew 500 ppm
I20260928 14:39:59.026176 26186 webserver.cc:460] Webserver started at http://127.6.23.194:33109/ using document root <none> and password file <none>
I20260928 14:39:59.026772 26186 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:59.026834 26186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:59.027055 26186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:59.029201 26186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/data/instance:
uuid: "2b7862ebf1d34017842ab486607e7861"
format_stamp: "Formatted at 2026-09-28 14:39:59 on dist-test-slave-06lh"
server_key: "c2a42e1a631972864e89c5b2c4ae72fd"
server_key_iv: "f954d12dc9e65bcf93e2a512aa8fc85d"
server_key_version: "encryptionkey@0"
I20260928 14:39:59.029738 26186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance:
uuid: "2b7862ebf1d34017842ab486607e7861"
format_stamp: "Formatted at 2026-09-28 14:39:59 on dist-test-slave-06lh"
server_key: "c2a42e1a631972864e89c5b2c4ae72fd"
server_key_iv: "f954d12dc9e65bcf93e2a512aa8fc85d"
server_key_version: "encryptionkey@0"
I20260928 14:39:59.033354 26186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:39:59.035785 26200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:59.037002 26186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:39:59.037139 26186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/wal
uuid: "2b7862ebf1d34017842ab486607e7861"
format_stamp: "Formatted at 2026-09-28 14:39:59 on dist-test-slave-06lh"
server_key: "c2a42e1a631972864e89c5b2c4ae72fd"
server_key_iv: "f954d12dc9e65bcf93e2a512aa8fc85d"
server_key_version: "encryptionkey@0"
I20260928 14:39:59.037285 26186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:59.053165 26186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:59.054289 26186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:59.054488 26186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:59.055500 26186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:59.057003 26186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:59.057063 26186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:59.057127 26186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:59.057163 26186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:59.070815 26186 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.194:44157
I20260928 14:39:59.070847 26313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.194:44157 every 8 connection(s)
I20260928 14:39:59.072023 26186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/data/info.pb
I20260928 14:39:59.077947 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 26186
I20260928 14:39:59.078083 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/wal/instance
I20260928 14:39:59.078423 6239 external_mini_cluster.cc:1428] Setting key e88e0430493358ac64a3ef98ee8458d7
I20260928 14:39:59.080421 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.23.195:0
--local_ip_for_outbound_sockets=127.6.23.195
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34813
--builtin_ntp_servers=127.6.23.212:38801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:39:59.084545 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.074151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:36621 (local address 127.6.23.254:34813)
0928 14:39:59.074297 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.074303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.075751 (+ 1448us) server_negotiation.cc:407] Connection header received
0928 14:39:59.077813 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.077817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.077883 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.077944 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:59.079452 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.080406 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.082285 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.082517 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.082628 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.083884 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:59.083890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:59.083895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:59.083911 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:59.083942 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:59.083955 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:59.084052 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.084256 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.084342 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:39:59.085752 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.074709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34813 (local address 127.6.23.194:36621)
0928 14:39:59.075579 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.075634 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:59.077538 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:59.078446 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:59.078457 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:59.078573 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:59.079291 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.079311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.080688 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.080694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.082153 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.082169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.082665 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.082669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.082863 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.083570 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:59.083574 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:59.083627 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:59.083710 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:59.084062 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:59.084074 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:59.084083 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:39:59.084211 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 14:39:59.084520 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":143,"threads_started":1}
I20260928 14:39:59.088482 26314 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34813
I20260928 14:39:59.088773 26314 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:59.089457 26314 heartbeater.cc:511] Master 127.6.23.254:34813 requested a full tablet report, sending...
I20260928 14:39:59.090729 25995 ts_manager.cc:194] Registered new tserver with Master: 2b7862ebf1d34017842ab486607e7861 (127.6.23.194:44157)
I20260928 14:39:59.091665 25995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.194:36621
W20260928 14:39:59.195600 26317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:39:59.195879 26317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:39:59.195945 26317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:39:59.199949 26317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:39:59.200024 26317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:39:59.200116 26317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.23.195
I20260928 14:39:59.204926 26317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:38801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.23.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.23.195
--webserver_port=0
--tserver_master_addrs=127.6.23.254:34813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.23.195
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:39:59.206017 26317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:39:59.207454 26317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:39:59.216830 26323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:59.216849 26322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:39:59.216849 26325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:39:59.217337 26317 server_base.cc:1034] running on GCE node
I20260928 14:39:59.217684 26317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:39:59.218313 26317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:39:59.219493 26317 hybrid_clock.cc:648] HybridClock initialized: now 1790606399219477 us; error 39 us; skew 500 ppm
I20260928 14:39:59.225200 26317 webserver.cc:460] Webserver started at http://127.6.23.195:42295/ using document root <none> and password file <none>
I20260928 14:39:59.225836 26317 fs_manager.cc:359] Metadata directory not provided
I20260928 14:39:59.225934 26317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:39:59.226176 26317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:39:59.228402 26317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/data/instance:
uuid: "49d83099f4ab43118b590deb375b82ef"
format_stamp: "Formatted at 2026-09-28 14:39:59 on dist-test-slave-06lh"
server_key: "e0a54fe197e1927bed12e8a26d86e811"
server_key_iv: "d0b1da6172a3c719d860ecabf8b5667f"
server_key_version: "encryptionkey@0"
I20260928 14:39:59.229003 26317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance:
uuid: "49d83099f4ab43118b590deb375b82ef"
format_stamp: "Formatted at 2026-09-28 14:39:59 on dist-test-slave-06lh"
server_key: "e0a54fe197e1927bed12e8a26d86e811"
server_key_iv: "d0b1da6172a3c719d860ecabf8b5667f"
server_key_version: "encryptionkey@0"
I20260928 14:39:59.232733 26317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:39:59.235091 26331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:59.236402 26317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:39:59.236553 26317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/wal
uuid: "49d83099f4ab43118b590deb375b82ef"
format_stamp: "Formatted at 2026-09-28 14:39:59 on dist-test-slave-06lh"
server_key: "e0a54fe197e1927bed12e8a26d86e811"
server_key_iv: "d0b1da6172a3c719d860ecabf8b5667f"
server_key_version: "encryptionkey@0"
I20260928 14:39:59.236668 26317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:39:59.266552 26317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:39:59.267383 26317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:39:59.267583 26317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:39:59.268194 26317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:39:59.269164 26317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:39:59.269234 26317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:59.269300 26317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:39:59.269343 26317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:39:59.277948 26317 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.195:38513
I20260928 14:39:59.277980 26444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.195:38513 every 8 connection(s)
I20260928 14:39:59.278849 26317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/data/info.pb
I20260928 14:39:59.287904 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.280645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.195:34943 (local address 127.6.23.254:34813)
0928 14:39:59.280781 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.280785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.281765 (+ 980us) server_negotiation.cc:407] Connection header received
0928 14:39:59.283158 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.283161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.283218 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.283362 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:59.284252 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.284798 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.285952 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.286184 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.286302 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.287417 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:59.287420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:59.287422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:59.287433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:59.287449 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:59.287464 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:59.287539 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.287694 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.287760 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:39:59.288379 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 26317
I20260928 14:39:59.288515 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/wal/instance
I20260928 14:39:59.288838 6239 external_mini_cluster.cc:1428] Setting key ca8f65cbbdcbb851c738c28847acc23b
I20260928 14:39:59.288973 26447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.280973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:34813 (local address 127.6.23.195:34943)
0928 14:39:59.281612 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.281651 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:59.282950 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:59.283542 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:59.283549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:59.283636 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:39:59.284112 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.284124 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.284944 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.284948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.285827 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.285837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.286343 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.286348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.286514 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.287075 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:39:59.287076 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:39:59.287114 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:39:59.287190 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:39:59.287539 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:39:59.287550 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:39:59.287558 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:39:59.287693 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 14:39:59.287964 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260928 14:39:59.290098 26445 heartbeater.cc:347] Connected to a master server at 127.6.23.254:34813
I20260928 14:39:59.290325 26445 heartbeater.cc:464] Registering TS with master...
I20260928 14:39:59.290920 26445 heartbeater.cc:511] Master 127.6.23.254:34813 requested a full tablet report, sending...
I20260928 14:39:59.291944 25995 ts_manager.cc:194] Registered new tserver with Master: 49d83099f4ab43118b590deb375b82ef (127.6.23.195:38513)
I20260928 14:39:59.292459 25995 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.23.195:34943
I20260928 14:39:59.302738 6239 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:39:59.308688 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.304840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.6.23.254:34813)
0928 14:39:59.304983 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.304988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.305169 (+ 181us) server_negotiation.cc:407] Connection header received
0928 14:39:59.305355 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.305358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.305404 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.305489 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:39:59.305761 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.306280 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.307124 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.307395 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.307503 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.308241 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:39:59.308245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:39:59.308248 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:39:59.308264 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:39:59.308285 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:39:59.308295 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:39:59.308365 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.308491 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.308547 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:39:59.311427 25995 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:39:59.314397 25995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:39:59.328835 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.323381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36904 (local address 127.6.23.194:44157)
0928 14:39:59.323976 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.323989 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.324016 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:39:59.324097 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.324101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.324318 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.324471 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:59.324811 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.325932 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.327542 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.328149 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.328273 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.328471 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.328571 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.328666 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":102,"threads_started":1}
I20260928 14:39:59.328830 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.322729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:44157 (local address 127.0.0.1:36904)
0928 14:39:59.323670 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.323691 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:59.323786 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:59.324456 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:59.324458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:59.324473 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:59.324554 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.324559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.326021 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.326027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.327406 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.327416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.328273 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.328284 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.328386 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.328423 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:39:59.328541 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 14:39:59.328666 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":25,"thread_start_us":86,"threads_started":1}
I20260928 14:39:59.329167 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.322711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.6.23.195:38513)
0928 14:39:59.323495 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.323507 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.323536 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:39:59.323787 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.323794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.324020 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.324181 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:59.324633 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.325861 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.327475 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.328200 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.328320 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.328502 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.328911 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.329007 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":120,"threads_started":1}
I20260928 14:39:59.329428 26457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.322969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:40407 (local address 127.0.0.1:42164)
0928 14:39:59.323964 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.323976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:59.324051 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:59.325181 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:59.325184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:59.325201 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:59.325280 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.325286 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.326496 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.326499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.327617 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.327625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.328801 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.328805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.328906 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.328929 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:39:59.329230 (+ 301us) client_negotiation.cc:240] Negotiation successful
0928 14:39:59.329304 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"thread_start_us":107,"threads_started":1}
I20260928 14:39:59.329594 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.323494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.6.23.193:40407)
0928 14:39:59.324785 (+ 1291us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.324817 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.324841 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:39:59.324907 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.324911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.325058 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.325210 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:59.325373 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.326248 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.327736 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.328464 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.328564 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.328744 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.329347 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.329443 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":124,"threads_started":1}
I20260928 14:39:59.330497 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.322570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:38513 (local address 127.0.0.1:36570)
0928 14:39:59.323370 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.323421 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:59.323559 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:59.324331 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:59.324338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:59.324361 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:59.324514 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.324521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.325980 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.325983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.327356 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.327366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.328565 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.328569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.328662 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.328685 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:39:59.328758 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:39:59.328835 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":52,"thread_start_us":71,"threads_started":1}
I20260928 14:39:59.332325 26117 tablet_service.cc:1467] Processing CreateTablet for tablet c6e3132e6e394c6682366deaf3a33c0c (DEFAULT_TABLE table=test-table [id=9678ddb1d77143659c2f31230cf52872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:59.332325 26246 tablet_service.cc:1467] Processing CreateTablet for tablet c6e3132e6e394c6682366deaf3a33c0c (DEFAULT_TABLE table=test-table [id=9678ddb1d77143659c2f31230cf52872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:59.332700 26379 tablet_service.cc:1467] Processing CreateTablet for tablet c6e3132e6e394c6682366deaf3a33c0c (DEFAULT_TABLE table=test-table [id=9678ddb1d77143659c2f31230cf52872]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:39:59.333451 26246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e3132e6e394c6682366deaf3a33c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:59.333451 26117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e3132e6e394c6682366deaf3a33c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:59.333716 26379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e3132e6e394c6682366deaf3a33c0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:39:59.339941 26462 tablet_bootstrap.cc:492] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Bootstrap starting.
I20260928 14:39:59.341300 26464 tablet_bootstrap.cc:492] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: Bootstrap starting.
I20260928 14:39:59.342495 26462 tablet_bootstrap.cc:654] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:59.343217 26462 log.cc:824] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:59.343379 26464 tablet_bootstrap.cc:654] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:59.343571 26463 tablet_bootstrap.cc:492] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8: Bootstrap starting.
I20260928 14:39:59.344228 26464 log.cc:824] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:59.345446 26462 tablet_bootstrap.cc:492] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: No bootstrap required, opened a new log
I20260928 14:39:59.345597 26463 tablet_bootstrap.cc:654] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8: Neither blocks nor log segments found. Creating new log.
I20260928 14:39:59.345634 26462 ts_tablet_manager.cc:1397] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:39:59.346292 26464 tablet_bootstrap.cc:492] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: No bootstrap required, opened a new log
I20260928 14:39:59.346310 26463 log.cc:824] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:39:59.346463 26464 ts_tablet_manager.cc:1397] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260928 14:39:59.348354 26463 tablet_bootstrap.cc:492] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8: No bootstrap required, opened a new log
I20260928 14:39:59.348553 26463 ts_tablet_manager.cc:1397] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:39:59.350039 26462 raft_consensus.cc:348] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.350348 26462 raft_consensus.cc:374] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:59.350409 26462 raft_consensus.cc:729] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49d83099f4ab43118b590deb375b82ef, State: Initialized, Role: FOLLOWER
I20260928 14:39:59.350617 26464 raft_consensus.cc:348] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.350853 26464 raft_consensus.cc:374] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:59.350906 26464 raft_consensus.cc:729] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7862ebf1d34017842ab486607e7861, State: Initialized, Role: FOLLOWER
I20260928 14:39:59.350854 26462 consensus_queue.cc:260] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.351379 26464 consensus_queue.cc:260] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.351751 26445 heartbeater.cc:503] Master 127.6.23.254:34813 was elected leader, sending a full tablet report...
I20260928 14:39:59.351881 26462 ts_tablet_manager.cc:1428] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:39:59.352061 26463 raft_consensus.cc:348] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.352276 26464 ts_tablet_manager.cc:1428] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:39:59.352289 26463 raft_consensus.cc:374] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:39:59.352340 26463 raft_consensus.cc:729] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03578579ee54495aa10765b3576e2d8, State: Initialized, Role: FOLLOWER
I20260928 14:39:59.352417 26314 heartbeater.cc:503] Master 127.6.23.254:34813 was elected leader, sending a full tablet report...
I20260928 14:39:59.352728 26463 consensus_queue.cc:260] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.353570 26183 heartbeater.cc:503] Master 127.6.23.254:34813 was elected leader, sending a full tablet report...
I20260928 14:39:59.353634 26463 ts_tablet_manager.cc:1428] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 14:39:59.369904 26184 tablet.cc:2369] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:39:59.529814 26446 tablet.cc:2369] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:39:59.573563 26315 tablet.cc:2369] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:39:59.633176 26469 raft_consensus.cc:482] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:39:59.633425 26469 raft_consensus.cc:504] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.634557 26469 leader_election.cc:288] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49d83099f4ab43118b590deb375b82ef (127.6.23.195:38513), f03578579ee54495aa10765b3576e2d8 (127.6.23.193:40407)
I20260928 14:39:59.640219 26461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.634976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:50101 (local address 127.6.23.193:40407)
0928 14:39:59.635139 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.635143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.635391 (+ 248us) server_negotiation.cc:407] Connection header received
0928 14:39:59.635472 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.635474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.635523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.635607 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:59.636074 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.636876 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.638726 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.639392 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.639514 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.639565 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.639981 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.640063 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:39:59.640219 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.635210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.23.194:38173 (local address 127.6.23.195:38513)
0928 14:39:59.635384 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.635389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.635403 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:39:59.635463 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.635466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.635519 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.635608 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:39:59.636074 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.636869 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.638726 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.639403 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.639520 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.639565 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.639980 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.640065 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 14:39:59.640363 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.634838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:40407 (local address 127.6.23.194:50101)
0928 14:39:59.635295 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.635307 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:59.635393 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:59.635663 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:59.635667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:59.635712 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:59.635914 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.635922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.636994 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.636997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.638578 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.638589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.639526 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.639530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.639799 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.639869 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 14:39:59.639949 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:39:59.640072 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":417,"spinlock_wait_cycles":28416,"thread_start_us":83,"threads_started":1}
I20260928 14:39:59.640705 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.634837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.195:38513 (local address 127.6.23.194:38173)
0928 14:39:59.635156 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.635173 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:39:59.635333 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:39:59.635663 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:39:59.635667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:39:59.635713 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:39:59.635918 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.635922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.636994 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.636997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.638578 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:39:59.638589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.639526 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.639530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:39:59.639794 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.639869 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 14:39:59.639949 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:39:59.640071 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":244,"spinlock_wait_cycles":18688,"thread_start_us":90,"threads_started":1}
I20260928 14:39:59.641021 26137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3132e6e394c6682366deaf3a33c0c" candidate_uuid: "2b7862ebf1d34017842ab486607e7861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03578579ee54495aa10765b3576e2d8" is_pre_election: true
I20260928 14:39:59.641352 26137 raft_consensus.cc:2454] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7862ebf1d34017842ab486607e7861 in term 0.
I20260928 14:39:59.641803 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3132e6e394c6682366deaf3a33c0c" candidate_uuid: "2b7862ebf1d34017842ab486607e7861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d83099f4ab43118b590deb375b82ef" is_pre_election: true
I20260928 14:39:59.641934 26201 leader_election.cc:302] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b7862ebf1d34017842ab486607e7861, f03578579ee54495aa10765b3576e2d8; no voters:
I20260928 14:39:59.642091 26399 raft_consensus.cc:2454] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b7862ebf1d34017842ab486607e7861 in term 0.
I20260928 14:39:59.642231 26469 raft_consensus.cc:2781] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:39:59.642304 26469 raft_consensus.cc:482] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:39:59.642333 26469 raft_consensus.cc:3037] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:59.643461 26469 raft_consensus.cc:504] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.643857 26469 leader_election.cc:288] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [CANDIDATE]: Term 1 election: Requested vote from peers 49d83099f4ab43118b590deb375b82ef (127.6.23.195:38513), f03578579ee54495aa10765b3576e2d8 (127.6.23.193:40407)
I20260928 14:39:59.644214 26399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3132e6e394c6682366deaf3a33c0c" candidate_uuid: "2b7862ebf1d34017842ab486607e7861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49d83099f4ab43118b590deb375b82ef"
I20260928 14:39:59.644270 26137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e3132e6e394c6682366deaf3a33c0c" candidate_uuid: "2b7862ebf1d34017842ab486607e7861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03578579ee54495aa10765b3576e2d8"
I20260928 14:39:59.644338 26399 raft_consensus.cc:3037] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:59.644372 26137 raft_consensus.cc:3037] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:39:59.645610 26399 raft_consensus.cc:2454] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7862ebf1d34017842ab486607e7861 in term 1.
I20260928 14:39:59.645610 26137 raft_consensus.cc:2454] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b7862ebf1d34017842ab486607e7861 in term 1.
I20260928 14:39:59.646015 26202 leader_election.cc:302] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2b7862ebf1d34017842ab486607e7861, 49d83099f4ab43118b590deb375b82ef, f03578579ee54495aa10765b3576e2d8; no voters:
I20260928 14:39:59.646157 26469 raft_consensus.cc:2781] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:39:59.646360 26469 raft_consensus.cc:686] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7862ebf1d34017842ab486607e7861, State: Running, Role: LEADER
I20260928 14:39:59.646698 26469 consensus_queue.cc:237] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } }
I20260928 14:39:59.649895 25991 catalog_manager.cc:5495] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b7862ebf1d34017842ab486607e7861 (127.6.23.194). New cstate: current_term: 1 leader_uuid: "2b7862ebf1d34017842ab486607e7861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b7862ebf1d34017842ab486607e7861" member_type: VOTER last_known_addr { host: "127.6.23.194" port: 44157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 } health_report { overall_health: UNKNOWN } } }
I20260928 14:39:59.693102 26460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.689889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36918 (local address 127.6.23.194:44157)
0928 14:39:59.690017 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:39:59.690020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:39:59.690033 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:39:59.690108 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:39:59.690110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:39:59.690165 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:39:59.690255 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:39:59.690484 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.690970 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.691797 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:39:59.692003 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:39:59.692081 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:39:59.692342 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:39:59.692808 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:39:59.692898 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:39:59.692949 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:39:59.692993 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:39:59.698949 26399 raft_consensus.cc:1261] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 1 FOLLOWER]: Refusing update from remote peer 2b7862ebf1d34017842ab486607e7861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:59.698949 26137 raft_consensus.cc:1261] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 1 FOLLOWER]: Refusing update from remote peer 2b7862ebf1d34017842ab486607e7861: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:39:59.699699 26473 consensus_queue.cc:1035] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03578579ee54495aa10765b3576e2d8" member_type: VOTER last_known_addr { host: "127.6.23.193" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:59.699846 26469 consensus_queue.cc:1035] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49d83099f4ab43118b590deb375b82ef" member_type: VOTER last_known_addr { host: "127.6.23.195" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:39:59.708715 26477 mvcc.cc:205] Tried to move back new op lower bound from 7334323813159788544 to 7334323812956119040. Current Snapshot: MvccSnapshot[applied={T|T < 7334323813159788544}]
I20260928 14:39:59.716444 25991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:59.716624 25991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:39:59.719033 25991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 452d8ae4e6b24087acdbe076ed3b6bc0: Sending DeleteTablet for 3 replicas of tablet c6e3132e6e394c6682366deaf3a33c0c
I20260928 14:39:59.719882 26379 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e3132e6e394c6682366deaf3a33c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:59 UTC) from {username='slave'} at 127.0.0.1:36570
I20260928 14:39:59.719901 26117 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e3132e6e394c6682366deaf3a33c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:59 UTC) from {username='slave'} at 127.0.0.1:42164
I20260928 14:39:59.720371 26486 tablet_replica.cc:330] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: stopping tablet replica
I20260928 14:39:59.720427 26246 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e3132e6e394c6682366deaf3a33c0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:39:59 UTC) from {username='slave'} at 127.0.0.1:36904
I20260928 14:39:59.720424 26487 tablet_replica.cc:330] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8: stopping tablet replica
I20260928 14:39:59.720798 26487 raft_consensus.cc:2229] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:59.720803 26486 raft_consensus.cc:2229] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:39:59.721128 26486 raft_consensus.cc:2258] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:59.721171 26487 raft_consensus.cc:2258] T c6e3132e6e394c6682366deaf3a33c0c P f03578579ee54495aa10765b3576e2d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:59.721607 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 26055
I20260928 14:39:59.722738 26486 ts_tablet_manager.cc:1905] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:59.724977 26488 tablet_replica.cc:330] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: stopping tablet replica
I20260928 14:39:59.725368 26488 raft_consensus.cc:2229] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:39:59.725999 26488 raft_consensus.cc:2258] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:39:59.726284 26486 ts_tablet_manager.cc:1918] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:59.726387 26486 log.cc:1196] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-2/wal/wals/c6e3132e6e394c6682366deaf3a33c0c
I20260928 14:39:59.726717 26486 ts_tablet_manager.cc:1939] T c6e3132e6e394c6682366deaf3a33c0c P 49d83099f4ab43118b590deb375b82ef: Deleting consensus metadata
I20260928 14:39:59.727795 26488 ts_tablet_manager.cc:1905] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:39:59.728076 25980 catalog_manager.cc:4841] TS 49d83099f4ab43118b590deb375b82ef (127.6.23.195:38513): tablet c6e3132e6e394c6682366deaf3a33c0c (table test-table [id=9678ddb1d77143659c2f31230cf52872]) successfully deleted
W20260928 14:39:59.730083 25979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.23.193:40407 (error 108)
I20260928 14:39:59.730604 26488 ts_tablet_manager.cc:1918] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:39:59.730691 26488 log.cc:1196] T c6e3132e6e394c6682366deaf3a33c0c P 2b7862ebf1d34017842ab486607e7861: Deleting WAL directory at /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606142807864-6239-0/minicluster-data/ts-1/wal/wals/c6e3132e6e394c6682366deaf3a33c0c
I20260928 14:39:59.730929 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 26186
I20260928 14:39:59.731107 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.730815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.193:40407 (local address 127.0.0.1:42170)
0928 14:39:59.730948 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.731026 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.193:40407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 14:39:59.732460 25979 catalog_manager.cc:4584] TS f03578579ee54495aa10765b3576e2d8 (127.6.23.193:40407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6e3132e6e394c6682366deaf3a33c0c: Network error: Client connection negotiation failed: client connection to 127.6.23.193:40407: connect: Connection refused (error 111)
I20260928 14:39:59.736292 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 26317
I20260928 14:39:59.738070 26459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:39:59.736481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.194:44157 (local address 127.0.0.1:36930)
0928 14:39:59.737894 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0928 14:39:59.737964 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.194:44157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1308}
I20260928 14:39:59.741788 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 25964
2026-09-28T14:39:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1323 ms)
[----------] 6 tests from EncryptionPolicyTest (10558 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:39:59 dist-test-slave-06lh krb5kdc[26492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:39:59 dist-test-slave-06lh krb5kdc[26492](info): set up 2 sockets
Sep 28 14:39:59 dist-test-slave-06lh krb5kdc[26492](info): commencing operation
krb5kdc: starting...
W20260928 14:40:00.771641 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:00 dist-test-slave-06lh krb5kdc[26492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:00.894833 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:01.004592 26505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:01.004877 26505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:01.004951 26505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:01.009025 26505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:01.009092 26505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:01.009112 26505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:01.009130 26505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:01.009145 26505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:01.014493 26505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:01.015662 26505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:01.016721 26505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:01.025209 26513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:01.025430 26511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:01.025449 26510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:01.025723 26505 server_base.cc:1034] running on GCE node
I20260928 14:40:01.026376 26505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:01.027549 26505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:01.028740 26505 hybrid_clock.cc:648] HybridClock initialized: now 1790606401028728 us; error 26 us; skew 500 ppm
Sep 28 14:40:01 dist-test-slave-06lh krb5kdc[26492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:01.031525 26505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:01.032352 26505 webserver.cc:460] Webserver started at http://127.0.0.1:36711/ using document root <none> and password file <none>
I20260928 14:40:01.032876 26505 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:01.032934 26505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:01.033094 26505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:01.035137 26505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "914d67a81a324c95a5a22a322145c65b"
format_stamp: "Formatted at 2026-09-28 14:40:01 on dist-test-slave-06lh"
server_key: "c87976fba4c13612ff88d069f9576d23"
server_key_iv: "5e39323b5e6b65df8502c9d1b0fa64a1"
server_key_version: "encryptionkey@0"
I20260928 14:40:01.035732 26505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "914d67a81a324c95a5a22a322145c65b"
format_stamp: "Formatted at 2026-09-28 14:40:01 on dist-test-slave-06lh"
server_key: "c87976fba4c13612ff88d069f9576d23"
server_key_iv: "5e39323b5e6b65df8502c9d1b0fa64a1"
server_key_version: "encryptionkey@0"
I20260928 14:40:01.039426 26505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:40:01.041919 26520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:01.043318 26505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:01.043440 26505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "914d67a81a324c95a5a22a322145c65b"
format_stamp: "Formatted at 2026-09-28 14:40:01 on dist-test-slave-06lh"
server_key: "c87976fba4c13612ff88d069f9576d23"
server_key_iv: "5e39323b5e6b65df8502c9d1b0fa64a1"
server_key_version: "encryptionkey@0"
I20260928 14:40:01.043550 26505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:01.061549 26505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:01.064591 26505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:01.064805 26505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:01.071910 26505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40505
I20260928 14:40:01.071938 26572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40505 every 8 connection(s)
I20260928 14:40:01.072875 26505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:01.075698 26573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:01.081178 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 26505
I20260928 14:40:01.081318 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:01.081352 26573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b: Bootstrap starting.
I20260928 14:40:01.081629 6239 external_mini_cluster.cc:1428] Setting key e2535cd18eeb1c38d5a2fa43d37d4709
I20260928 14:40:01.083966 26573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:01.084698 26573 log.cc:824] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:01.086547 26573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b: No bootstrap required, opened a new log
I20260928 14:40:01.090183 26573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914d67a81a324c95a5a22a322145c65b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260928 14:40:01.090411 26573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:01.090474 26573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d67a81a324c95a5a22a322145c65b, State: Initialized, Role: FOLLOWER
Sep 28 14:40:01 dist-test-slave-06lh krb5kdc[26492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:40:01.090983 26573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914d67a81a324c95a5a22a322145c65b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260928 14:40:01.091140 26573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:01.091234 26573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:01.091367 26573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:01.092532 26573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914d67a81a324c95a5a22a322145c65b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260928 14:40:01.092882 26573 leader_election.cc:302] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 914d67a81a324c95a5a22a322145c65b; no voters:
I20260928 14:40:01.093217 26573 leader_election.cc:288] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:01.093336 26578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:01.093544 26578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [term 1 LEADER]: Becoming Leader. State: Replica: 914d67a81a324c95a5a22a322145c65b, State: Running, Role: LEADER
I20260928 14:40:01.093837 26578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914d67a81a324c95a5a22a322145c65b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } }
I20260928 14:40:01.094651 26573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:01.095010 26580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 914d67a81a324c95a5a22a322145c65b. Latest consensus state: current_term: 1 leader_uuid: "914d67a81a324c95a5a22a322145c65b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914d67a81a324c95a5a22a322145c65b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } } }
I20260928 14:40:01.095142 26580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:01.095440 26579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "914d67a81a324c95a5a22a322145c65b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "914d67a81a324c95a5a22a322145c65b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40505 } } }
I20260928 14:40:01.095530 26579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:01.097285 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:01.083039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59081 (local address 127.0.0.1:40505)
0928 14:40:01.083496 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:01.083517 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:01.083552 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:40:01.084276 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:01.084295 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:01.084679 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:01.085066 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:01.085926 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:01.086945 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:01.087876 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:01.088354 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:01.088526 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:01.091213 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:01.091222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:01.091259 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:01.091295 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:01.093248 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:01.093841 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:01.093849 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:01.093857 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:01.093997 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:01.094414 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:01.094418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:01.094420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:01.094918 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:01.095151 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:01.095974 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 14:40:01.096224 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":157,"threads_started":1}
W20260928 14:40:01.099799 26593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:01.099879 26593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:01.099970 26582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:01.100319 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:01.100795 26582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:01.106276 26582 catalog_manager.cc:1347] Generated new cluster ID: 1ba638d52e0c4959b7cbadaf2cc25ce6
I20260928 14:40:01.106338 26582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:01.115738 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:01.106434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46825 (local address 127.0.0.1:40505)
0928 14:40:01.106558 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:01.106563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:01.106981 (+ 418us) server_negotiation.cc:407] Connection header received
0928 14:40:01.107119 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:01.107123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:01.107214 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:01.107326 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:01.108150 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:01.108955 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:01.110107 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:01.110469 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:01.110555 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:01.112238 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:01.112242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:01.112245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:01.112268 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:01.113933 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:01.114466 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:01.114469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:01.114471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:01.114550 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:01.114882 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:01.114884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:01.114886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:01.115069 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:01.115200 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:01.115493 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:40:01.115585 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:40:01.125669 26582 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:01.126726 26582 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:01.135689 26582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 914d67a81a324c95a5a22a322145c65b: Generated new TSK 0
I20260928 14:40:01.136476 26582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:01.142397 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 26505
W20260928 14:40:01.148175 26490 connection.cc:535] client connection to 127.0.0.1:40505 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40505): Connection reset by peer (error 104)
2026-09-28T14:40:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:01 dist-test-slave-06lh krb5kdc[26604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:01 dist-test-slave-06lh krb5kdc[26604](info): set up 2 sockets
Sep 28 14:40:01 dist-test-slave-06lh krb5kdc[26604](info): commencing operation
krb5kdc: starting...
W20260928 14:40:02.179903 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:02 dist-test-slave-06lh krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:02.306061 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:02.419574 26617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:02.419860 26617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:02.419903 26617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:02.423894 26617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:02.423959 26617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:02.423979 26617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:02.423996 26617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:02.424011 26617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:02.429422 26617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:02.430523 26617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:02.431629 26617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:02.440460 26625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:02.440483 26623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:02.440469 26622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:02.440716 26617 server_base.cc:1034] running on GCE node
I20260928 14:40:02.441362 26617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:02.442508 26617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:02.443667 26617 hybrid_clock.cc:648] HybridClock initialized: now 1790606402443654 us; error 24 us; skew 500 ppm
Sep 28 14:40:02 dist-test-slave-06lh krb5kdc[26604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:02.446502 26617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:02.447542 26617 webserver.cc:460] Webserver started at http://127.0.0.1:44103/ using document root <none> and password file <none>
I20260928 14:40:02.448129 26617 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:02.448221 26617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:02.448468 26617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:02.450594 26617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "03b7ba60f9264754af66eba8138862ef"
format_stamp: "Formatted at 2026-09-28 14:40:02 on dist-test-slave-06lh"
server_key: "193cd9e0c175feb419a1ee8500407f3d"
server_key_iv: "372e92240a7d82b0f7dcb643d9b39255"
server_key_version: "encryptionkey@0"
I20260928 14:40:02.451191 26617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "03b7ba60f9264754af66eba8138862ef"
format_stamp: "Formatted at 2026-09-28 14:40:02 on dist-test-slave-06lh"
server_key: "193cd9e0c175feb419a1ee8500407f3d"
server_key_iv: "372e92240a7d82b0f7dcb643d9b39255"
server_key_version: "encryptionkey@0"
I20260928 14:40:02.455118 26617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:40:02.457684 26632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:02.458969 26617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:02.459128 26617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "03b7ba60f9264754af66eba8138862ef"
format_stamp: "Formatted at 2026-09-28 14:40:02 on dist-test-slave-06lh"
server_key: "193cd9e0c175feb419a1ee8500407f3d"
server_key_iv: "372e92240a7d82b0f7dcb643d9b39255"
server_key_version: "encryptionkey@0"
I20260928 14:40:02.459280 26617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:02.470752 26617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:02.473769 26617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:02.473985 26617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:02.480958 26617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39651
I20260928 14:40:02.481025 26684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39651 every 8 connection(s)
I20260928 14:40:02.481936 26617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:02.482549 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 26617
I20260928 14:40:02.482717 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:02.483031 6239 external_mini_cluster.cc:1428] Setting key 3316f3caeb5fd49e338bc4af2a6a5517
I20260928 14:40:02.485430 26685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:02.492048 26685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef: Bootstrap starting.
I20260928 14:40:02.492641 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:02.484724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40689 (local address 127.0.0.1:39651)
0928 14:40:02.485166 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:02.485184 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:02.485220 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:40:02.485924 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:02.485942 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:02.486281 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:02.486565 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:02.486979 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:02.488051 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:02.489257 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:02.489719 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:02.489842 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:02.490764 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:02.490769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:02.490781 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:02.491169 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:02.491269 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:02.491312 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:02.491419 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:02.491624 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 14:40:02.491792 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":146,"threads_started":1}
I20260928 14:40:02.494517 26685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:02.495424 26685 log.cc:824] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:02.495712 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:02.497697 26685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef: No bootstrap required, opened a new log
I20260928 14:40:02.501518 26685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b7ba60f9264754af66eba8138862ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39651 } }
I20260928 14:40:02.501791 26685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:02.501892 26685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b7ba60f9264754af66eba8138862ef, State: Initialized, Role: FOLLOWER
I20260928 14:40:02.502373 26685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b7ba60f9264754af66eba8138862ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39651 } }
I20260928 14:40:02.502532 26685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:02.502605 26685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:02.502707 26685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:02.503639 26688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:02.497848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34055 (local address 127.0.0.1:39651)
0928 14:40:02.498060 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:02.498066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:02.498736 (+ 670us) server_negotiation.cc:407] Connection header received
0928 14:40:02.498989 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:02.498992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:02.499079 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:02.499173 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:02.499634 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:02.500892 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:02.501735 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:02.502089 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:02.502201 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:02.503118 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:02.503121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:02.503123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:02.503135 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:02.503160 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:02.503169 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:02.503295 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:02.503429 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:40:02.503495 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 14:40:02.504093 26685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b7ba60f9264754af66eba8138862ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39651 } }
I20260928 14:40:02.504474 26685 leader_election.cc:302] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03b7ba60f9264754af66eba8138862ef; no voters:
I20260928 14:40:02.504866 26685 leader_election.cc:288] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:02.504936 26695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:02.505116 26695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [term 1 LEADER]: Becoming Leader. State: Replica: 03b7ba60f9264754af66eba8138862ef, State: Running, Role: LEADER
I20260928 14:40:02.505434 26695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b7ba60f9264754af66eba8138862ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39651 } }
I20260928 14:40:02.506233 26685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:02.507278 26697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03b7ba60f9264754af66eba8138862ef. Latest consensus state: current_term: 1 leader_uuid: "03b7ba60f9264754af66eba8138862ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b7ba60f9264754af66eba8138862ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39651 } } }
I20260928 14:40:02.507431 26697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:02.507788 26696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03b7ba60f9264754af66eba8138862ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03b7ba60f9264754af66eba8138862ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39651 } } }
I20260928 14:40:02.507858 26704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:02.507946 26696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:02.511265 26704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:02.516768 26704 catalog_manager.cc:1347] Generated new cluster ID: a615878773ca45b6893f897dd5e784f3
I20260928 14:40:02.516842 26704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:02.534518 26704 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:02.535988 26704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:02.544740 26704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03b7ba60f9264754af66eba8138862ef: Generated new TSK 0
I20260928 14:40:02.545737 26704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:02.560523 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 26617
W20260928 14:40:02.566210 26602 connection.cc:535] client connection to 127.0.0.1:39651 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39651): Connection reset by peer (error 104)
2026-09-28T14:40:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:02 dist-test-slave-06lh krb5kdc[26716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:02 dist-test-slave-06lh krb5kdc[26716](info): set up 2 sockets
Sep 28 14:40:02 dist-test-slave-06lh krb5kdc[26716](info): commencing operation
krb5kdc: starting...
W20260928 14:40:03.603737 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:03 dist-test-slave-06lh krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:03.731025 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:03.845511 26729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:03.845854 26729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:03.845952 26729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:03.850060 26729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:03.850164 26729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:03.850205 26729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:03.850266 26729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:03.850315 26729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:03.855770 26729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43957
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:03.856904 26729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:03.858007 26729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:03.866657 26737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:03.866880 26729 server_base.cc:1034] running on GCE node
W20260928 14:40:03.866654 26734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:03.866663 26735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:03.867798 26729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:03.869022 26729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:03.870224 26729 hybrid_clock.cc:648] HybridClock initialized: now 1790606403870209 us; error 32 us; skew 500 ppm
Sep 28 14:40:03 dist-test-slave-06lh krb5kdc[26716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:03.873180 26729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:03.874116 26729 webserver.cc:460] Webserver started at http://127.0.0.1:34683/ using document root <none> and password file <none>
I20260928 14:40:03.874655 26729 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:03.874716 26729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:03.874888 26729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:03.876998 26729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "95fd23469803447b8a30e209b2d74487"
format_stamp: "Formatted at 2026-09-28 14:40:03 on dist-test-slave-06lh"
server_key: "3798b609a8986f873f6aec6bb2a0a6cd"
server_key_iv: "7611d036c19300d1469537b77af1fe33"
server_key_version: "encryptionkey@0"
I20260928 14:40:03.877523 26729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "95fd23469803447b8a30e209b2d74487"
format_stamp: "Formatted at 2026-09-28 14:40:03 on dist-test-slave-06lh"
server_key: "3798b609a8986f873f6aec6bb2a0a6cd"
server_key_iv: "7611d036c19300d1469537b77af1fe33"
server_key_version: "encryptionkey@0"
I20260928 14:40:03.881273 26729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 14:40:03.883764 26744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:03.885010 26729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:03.885144 26729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "95fd23469803447b8a30e209b2d74487"
format_stamp: "Formatted at 2026-09-28 14:40:03 on dist-test-slave-06lh"
server_key: "3798b609a8986f873f6aec6bb2a0a6cd"
server_key_iv: "7611d036c19300d1469537b77af1fe33"
server_key_version: "encryptionkey@0"
I20260928 14:40:03.885258 26729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:03.889938 26729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:03.893286 26729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:03.893499 26729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:03.900513 26729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43957
I20260928 14:40:03.900540 26796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43957 every 8 connection(s)
I20260928 14:40:03.901484 26729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:03.904359 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:03.907580 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 26729
I20260928 14:40:03.907725 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:03.908013 6239 external_mini_cluster.cc:1428] Setting key 1db29c2382b245ad1540c641988a8ce7
I20260928 14:40:03.910542 26797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487: Bootstrap starting.
I20260928 14:40:03.913349 26797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:03.913713 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:03.909460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45381 (local address 127.0.0.1:43957)
0928 14:40:03.909926 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:03.909944 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:03.909985 (+ 41us) server_negotiation.cc:407] Connection header received
0928 14:40:03.910812 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:03.910835 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:03.911207 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:03.911595 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:03.912309 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:03.912315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:03.912329 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:03.912340 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:03.912399 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:03.912425 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:03.912556 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:03.912622 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 14:40:03.912783 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":122,"threads_started":1}
I20260928 14:40:03.914202 26797 log.cc:824] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:03.916324 26797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487: No bootstrap required, opened a new log
I20260928 14:40:03.916604 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:03.920473 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:03.918703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52395 (local address 127.0.0.1:43957)
0928 14:40:03.918844 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:03.918850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:03.919010 (+ 160us) server_negotiation.cc:407] Connection header received
0928 14:40:03.919134 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:03.919136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:03.919165 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:03.919267 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:03.920051 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:03.920054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:03.920055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:03.920065 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:03.920088 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:03.920097 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:03.920164 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:03.920286 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 14:40:03.920321 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:40:03.920807 26797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fd23469803447b8a30e209b2d74487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43957 } }
I20260928 14:40:03.921064 26797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:03.921124 26797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fd23469803447b8a30e209b2d74487, State: Initialized, Role: FOLLOWER
I20260928 14:40:03.921573 26797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fd23469803447b8a30e209b2d74487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43957 } }
I20260928 14:40:03.921720 26797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:03.921792 26797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:03.921900 26797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:03.922890 26797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fd23469803447b8a30e209b2d74487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43957 } }
I20260928 14:40:03.923290 26797 leader_election.cc:302] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95fd23469803447b8a30e209b2d74487; no voters:
I20260928 14:40:03.923643 26797 leader_election.cc:288] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:03.923743 26807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:03.923971 26807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [term 1 LEADER]: Becoming Leader. State: Replica: 95fd23469803447b8a30e209b2d74487, State: Running, Role: LEADER
I20260928 14:40:03.924357 26807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fd23469803447b8a30e209b2d74487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43957 } }
I20260928 14:40:03.924999 26797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:03.926218 26809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95fd23469803447b8a30e209b2d74487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fd23469803447b8a30e209b2d74487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43957 } } }
I20260928 14:40:03.926369 26809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:03.926638 26808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95fd23469803447b8a30e209b2d74487. Latest consensus state: current_term: 1 leader_uuid: "95fd23469803447b8a30e209b2d74487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95fd23469803447b8a30e209b2d74487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43957 } } }
I20260928 14:40:03.926735 26808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:03.926755 26816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:03.929860 26816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:03.935420 26816 catalog_manager.cc:1347] Generated new cluster ID: 336ebe6a18bf493483dadfe2834efd54
I20260928 14:40:03.935497 26816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:03.953922 26816 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:03.954298 26816 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:03.959184 26816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95fd23469803447b8a30e209b2d74487: Generated new TSK 0
I20260928 14:40:03.959947 26816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:03.966434 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 26729
2026-09-28T14:40:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:04 dist-test-slave-06lh krb5kdc[26828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:04 dist-test-slave-06lh krb5kdc[26828](info): set up 2 sockets
Sep 28 14:40:04 dist-test-slave-06lh krb5kdc[26828](info): commencing operation
krb5kdc: starting...
W20260928 14:40:05.007833 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:05 dist-test-slave-06lh krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:05.135567 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:05.247558 26841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:05.247833 26841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:05.247875 26841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:05.252265 26841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:05.252331 26841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:05.252351 26841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:05.252368 26841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:05.252426 26841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:05.257691 26841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:05.258843 26841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:05.260000 26841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:05.268625 26849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:05.268646 26846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:05.268651 26847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:05.268738 26841 server_base.cc:1034] running on GCE node
I20260928 14:40:05.269420 26841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:05.270493 26841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:05.271672 26841 hybrid_clock.cc:648] HybridClock initialized: now 1790606405271655 us; error 32 us; skew 500 ppm
Sep 28 14:40:05 dist-test-slave-06lh krb5kdc[26828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:05.280014 26841 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:05.280987 26841 webserver.cc:460] Webserver started at http://127.0.0.1:38559/ using document root <none> and password file <none>
I20260928 14:40:05.281625 26841 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:05.281709 26841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:05.281950 26841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:05.284140 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "f312ff7dffbb4427b5fcaa0021002082"
format_stamp: "Formatted at 2026-09-28 14:40:05 on dist-test-slave-06lh"
server_key: "cedd628ec68bb4ef991e553bc99b3a5b"
server_key_iv: "e578c275c2ca4f733a308eb10a3d6975"
server_key_version: "encryptionkey@0"
I20260928 14:40:05.284701 26841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "f312ff7dffbb4427b5fcaa0021002082"
format_stamp: "Formatted at 2026-09-28 14:40:05 on dist-test-slave-06lh"
server_key: "cedd628ec68bb4ef991e553bc99b3a5b"
server_key_iv: "e578c275c2ca4f733a308eb10a3d6975"
server_key_version: "encryptionkey@0"
I20260928 14:40:05.288486 26841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:40:05.291105 26856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:05.292505 26841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:40:05.292654 26841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "f312ff7dffbb4427b5fcaa0021002082"
format_stamp: "Formatted at 2026-09-28 14:40:05 on dist-test-slave-06lh"
server_key: "cedd628ec68bb4ef991e553bc99b3a5b"
server_key_iv: "e578c275c2ca4f733a308eb10a3d6975"
server_key_version: "encryptionkey@0"
I20260928 14:40:05.292786 26841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:05.312445 26841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:05.315343 26841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:05.315547 26841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:05.322923 26841 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45401
I20260928 14:40:05.322953 26908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45401 every 8 connection(s)
I20260928 14:40:05.323930 26841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:05.326804 26909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:05.332616 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 26841
I20260928 14:40:05.332787 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:05.332834 26909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082: Bootstrap starting.
I20260928 14:40:05.333110 6239 external_mini_cluster.cc:1428] Setting key e4f748a4eca19ec5b3347f11e3b11071
I20260928 14:40:05.335448 26909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:05.336187 26909 log.cc:824] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:05.338644 26909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082: No bootstrap required, opened a new log
I20260928 14:40:05.343006 26909 raft_consensus.cc:348] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f312ff7dffbb4427b5fcaa0021002082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45401 } }
I20260928 14:40:05.343325 26909 raft_consensus.cc:374] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:05.343391 26909 raft_consensus.cc:729] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f312ff7dffbb4427b5fcaa0021002082, State: Initialized, Role: FOLLOWER
Sep 28 14:40:05 dist-test-slave-06lh krb5kdc[26828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:40:05.343925 26909 consensus_queue.cc:260] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f312ff7dffbb4427b5fcaa0021002082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45401 } }
I20260928 14:40:05.344082 26909 raft_consensus.cc:388] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:05.344153 26909 raft_consensus.cc:482] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:05.344259 26909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:05.345647 26909 raft_consensus.cc:504] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f312ff7dffbb4427b5fcaa0021002082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45401 } }
I20260928 14:40:05.346040 26909 leader_election.cc:302] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f312ff7dffbb4427b5fcaa0021002082; no voters:
I20260928 14:40:05.346431 26909 leader_election.cc:288] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:05.346565 26914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:05.346803 26914 raft_consensus.cc:686] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [term 1 LEADER]: Becoming Leader. State: Replica: f312ff7dffbb4427b5fcaa0021002082, State: Running, Role: LEADER
I20260928 14:40:05.347164 26914 consensus_queue.cc:237] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f312ff7dffbb4427b5fcaa0021002082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45401 } }
I20260928 14:40:05.347911 26909 sys_catalog.cc:564] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:05.348320 26914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f312ff7dffbb4427b5fcaa0021002082. Latest consensus state: current_term: 1 leader_uuid: "f312ff7dffbb4427b5fcaa0021002082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f312ff7dffbb4427b5fcaa0021002082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45401 } } }
I20260928 14:40:05.348444 26914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:05.349152 26916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f312ff7dffbb4427b5fcaa0021002082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f312ff7dffbb4427b5fcaa0021002082" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45401 } } }
I20260928 14:40:05.349260 26916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:05.349509 26918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:05.350124 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:05.334606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58923 (local address 127.0.0.1:45401)
0928 14:40:05.335040 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:05.335056 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:05.335091 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:40:05.335855 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:05.335875 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:05.336229 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:05.336615 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:05.337551 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:05.339272 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:05.340412 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:05.341125 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:05.341343 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:05.344146 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:05.344153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:05.344184 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:05.344221 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:05.346303 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:05.346913 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:05.346919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:05.346925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:05.347066 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:05.347442 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:05.347445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:05.347447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:05.347970 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:05.348217 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:05.348521 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:40:05.349034 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":140,"threads_started":1}
I20260928 14:40:05.353080 26918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:05.353711 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:05.360937 26918 catalog_manager.cc:1347] Generated new cluster ID: a6d0c15c8e3244ce841572e68692751b
I20260928 14:40:05.361058 26918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:05.364909 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:05.355999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54171 (local address 127.0.0.1:45401)
0928 14:40:05.356121 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:05.356125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:05.356378 (+ 253us) server_negotiation.cc:407] Connection header received
0928 14:40:05.356532 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:05.356536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:05.356579 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:05.356673 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:05.357505 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:05.358320 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:05.359383 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:05.359775 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:05.359889 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:05.361508 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:05.361512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:05.361515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:05.361546 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:05.363062 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:05.363554 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:05.363558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:05.363559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:05.363642 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:05.363937 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:05.363941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:05.363943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:05.364121 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:05.364257 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:05.364507 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 14:40:05.364620 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:40:05.379395 26918 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:05.380379 26918 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:05.386659 26918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f312ff7dffbb4427b5fcaa0021002082: Generated new TSK 0
I20260928 14:40:05.387653 26918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:05.397110 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 26841
2026-09-28T14:40:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:05 dist-test-slave-06lh krb5kdc[26940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:05 dist-test-slave-06lh krb5kdc[26940](info): set up 2 sockets
Sep 28 14:40:05 dist-test-slave-06lh krb5kdc[26940](info): commencing operation
krb5kdc: starting...
W20260928 14:40:06.443778 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:06 dist-test-slave-06lh krb5kdc[26940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:06.569943 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:06.685761 26953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:06.686105 26953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:06.686184 26953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:06.690588 26953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:06.690693 26953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:06.690733 26953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:06.690773 26953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:06.690819 26953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:06.696434 26953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:06.697587 26953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:06.698664 26953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:06.707499 26958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:06.707499 26961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:06.707620 26959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:06.707685 26953 server_base.cc:1034] running on GCE node
I20260928 14:40:06.708644 26953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:06.709885 26953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:06.711057 26953 hybrid_clock.cc:648] HybridClock initialized: now 1790606406711039 us; error 32 us; skew 500 ppm
Sep 28 14:40:06 dist-test-slave-06lh krb5kdc[26940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:06.714130 26953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:06.715129 26953 webserver.cc:460] Webserver started at http://127.0.0.1:33847/ using document root <none> and password file <none>
I20260928 14:40:06.715739 26953 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:06.715802 26953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:06.716042 26953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:06.718204 26953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "63b4269533df42ecbb43ecce54a2a3e9"
format_stamp: "Formatted at 2026-09-28 14:40:06 on dist-test-slave-06lh"
server_key: "217a74622717571570766c5d5958ccd4"
server_key_iv: "5816a8f2574de945ff184cb1c7cd564e"
server_key_version: "encryptionkey@0"
I20260928 14:40:06.718783 26953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "63b4269533df42ecbb43ecce54a2a3e9"
format_stamp: "Formatted at 2026-09-28 14:40:06 on dist-test-slave-06lh"
server_key: "217a74622717571570766c5d5958ccd4"
server_key_iv: "5816a8f2574de945ff184cb1c7cd564e"
server_key_version: "encryptionkey@0"
I20260928 14:40:06.722671 26953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:40:06.725350 26968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:06.726639 26953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 14:40:06.726774 26953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "63b4269533df42ecbb43ecce54a2a3e9"
format_stamp: "Formatted at 2026-09-28 14:40:06 on dist-test-slave-06lh"
server_key: "217a74622717571570766c5d5958ccd4"
server_key_iv: "5816a8f2574de945ff184cb1c7cd564e"
server_key_version: "encryptionkey@0"
I20260928 14:40:06.726886 26953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:06.742419 26953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:06.745922 26953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:06.746146 26953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:06.753710 26953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34833
I20260928 14:40:06.753738 27020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34833 every 8 connection(s)
I20260928 14:40:06.754719 26953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:06.756547 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 26953
I20260928 14:40:06.756690 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:06.757002 6239 external_mini_cluster.cc:1428] Setting key 0b505e480d3d7d3f5a5c46777372e6fe
I20260928 14:40:06.757926 27021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:06.764062 27021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9: Bootstrap starting.
I20260928 14:40:06.766402 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:06.758519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43329 (local address 127.0.0.1:34833)
0928 14:40:06.758965 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:06.758980 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:06.759016 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:40:06.759699 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:06.759716 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:06.760055 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:06.760407 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:06.760818 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:06.761808 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:06.762766 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:06.763417 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:06.763585 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:06.764709 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:06.764715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:06.764728 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:06.765116 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:06.765170 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:06.765203 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:06.765315 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:06.765438 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 14:40:06.765663 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":147,"threads_started":1}
I20260928 14:40:06.766412 27021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:06.767298 27021 log.cc:824] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:06.769264 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:06.769405 27021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9: No bootstrap required, opened a new log
I20260928 14:40:06.773711 27021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b4269533df42ecbb43ecce54a2a3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } }
I20260928 14:40:06.774046 27021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:06.774113 27021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63b4269533df42ecbb43ecce54a2a3e9, State: Initialized, Role: FOLLOWER
I20260928 14:40:06.774617 27021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b4269533df42ecbb43ecce54a2a3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } }
I20260928 14:40:06.774796 27021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:06.774873 27021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:06.774997 27021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:06.775640 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:06.771326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44859 (local address 127.0.0.1:34833)
0928 14:40:06.771463 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:06.771468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:06.771659 (+ 191us) server_negotiation.cc:407] Connection header received
0928 14:40:06.771760 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:06.771765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:06.771812 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:06.771908 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:06.772144 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:06.772997 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:06.773792 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:06.774111 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:06.774224 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:06.775155 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:06.775158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:06.775161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:06.775173 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:06.775203 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:06.775222 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:06.775336 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:06.775422 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:40:06.775505 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:40:06.776269 27021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b4269533df42ecbb43ecce54a2a3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } }
I20260928 14:40:06.776674 27021 leader_election.cc:302] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63b4269533df42ecbb43ecce54a2a3e9; no voters:
I20260928 14:40:06.777048 27021 leader_election.cc:288] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:06.777174 27031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:06.777424 27031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [term 1 LEADER]: Becoming Leader. State: Replica: 63b4269533df42ecbb43ecce54a2a3e9, State: Running, Role: LEADER
I20260928 14:40:06.777752 27031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b4269533df42ecbb43ecce54a2a3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } }
I20260928 14:40:06.778447 27021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:06.779592 27032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63b4269533df42ecbb43ecce54a2a3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b4269533df42ecbb43ecce54a2a3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } } }
I20260928 14:40:06.779700 27032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:06.780027 27033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63b4269533df42ecbb43ecce54a2a3e9. Latest consensus state: current_term: 1 leader_uuid: "63b4269533df42ecbb43ecce54a2a3e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63b4269533df42ecbb43ecce54a2a3e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } } }
I20260928 14:40:06.780117 27033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:06.780124 27040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:06.783164 27040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:06.791203 27040 catalog_manager.cc:1347] Generated new cluster ID: ff8b7dec33f4434794d2ee47463d9cc0
I20260928 14:40:06.791309 27040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:06.828760 27040 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:06.830096 27040 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:06.846035 27040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63b4269533df42ecbb43ecce54a2a3e9: Generated new TSK 0
I20260928 14:40:06.846951 27040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:06.855142 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 26953
2026-09-28T14:40:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:06 dist-test-slave-06lh krb5kdc[27052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:06 dist-test-slave-06lh krb5kdc[27052](info): set up 2 sockets
Sep 28 14:40:06 dist-test-slave-06lh krb5kdc[27052](info): commencing operation
krb5kdc: starting...
W20260928 14:40:07.891711 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:07 dist-test-slave-06lh krb5kdc[27052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:08.019670 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:08.134473 27065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:08.134783 27065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:08.134828 27065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:08.139142 27065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:08.139247 27065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:08.139307 27065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:08.139339 27065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:08.139395 27065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:08.145239 27065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:08.146435 27065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:08.147609 27065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:08.156615 27071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:08.156685 27073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:08.156827 27065 server_base.cc:1034] running on GCE node
W20260928 14:40:08.156615 27070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:08.157590 27065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:08.158860 27065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:08.160050 27065 hybrid_clock.cc:648] HybridClock initialized: now 1790606408160008 us; error 60 us; skew 500 ppm
Sep 28 14:40:08 dist-test-slave-06lh krb5kdc[27052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:08.162822 27065 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:08.163874 27065 webserver.cc:460] Webserver started at http://127.0.0.1:34821/ using document root <none> and password file <none>
I20260928 14:40:08.164467 27065 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:08.164559 27065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:08.164809 27065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:08.167157 27065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "53000b7e54304f879bbfaec0c6167910"
format_stamp: "Formatted at 2026-09-28 14:40:08 on dist-test-slave-06lh"
server_key: "785ea3ccbb502fdf78740d20b55f8bdb"
server_key_iv: "bff5c13770ec7d5337330d57dd516a03"
server_key_version: "encryptionkey@0"
I20260928 14:40:08.167829 27065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "53000b7e54304f879bbfaec0c6167910"
format_stamp: "Formatted at 2026-09-28 14:40:08 on dist-test-slave-06lh"
server_key: "785ea3ccbb502fdf78740d20b55f8bdb"
server_key_iv: "bff5c13770ec7d5337330d57dd516a03"
server_key_version: "encryptionkey@0"
I20260928 14:40:08.171685 27065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:40:08.174150 27080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:08.175413 27065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:08.175565 27065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "53000b7e54304f879bbfaec0c6167910"
format_stamp: "Formatted at 2026-09-28 14:40:08 on dist-test-slave-06lh"
server_key: "785ea3ccbb502fdf78740d20b55f8bdb"
server_key_iv: "bff5c13770ec7d5337330d57dd516a03"
server_key_version: "encryptionkey@0"
I20260928 14:40:08.175686 27065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:08.180310 27065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:08.183333 27065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:08.183521 27065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:08.190665 27065 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34003
I20260928 14:40:08.190692 27132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34003 every 8 connection(s)
I20260928 14:40:08.191679 27065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:08.194411 27133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:08.196524 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27065
I20260928 14:40:08.196682 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:08.197005 6239 external_mini_cluster.cc:1428] Setting key 527489e6917a05f5525e270a9f75a1f1
I20260928 14:40:08.200891 27133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910: Bootstrap starting.
I20260928 14:40:08.203083 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:08.198541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57711 (local address 127.0.0.1:34003)
0928 14:40:08.199072 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:08.199089 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:08.199130 (+ 41us) server_negotiation.cc:407] Connection header received
0928 14:40:08.200055 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:08.200076 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:08.200442 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:08.200839 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:08.201750 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:08.201755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:08.201769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:08.201780 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:08.201838 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:08.201859 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:08.202021 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:08.202099 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 14:40:08.202335 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":169,"threads_started":1}
I20260928 14:40:08.203513 27133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:08.204388 27133 log.cc:824] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:08.206260 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:08.206594 27133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910: No bootstrap required, opened a new log
I20260928 14:40:08.210449 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:08.208647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46401 (local address 127.0.0.1:34003)
0928 14:40:08.208795 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:08.208800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:08.208997 (+ 197us) server_negotiation.cc:407] Connection header received
0928 14:40:08.209127 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:08.209129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:08.209162 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:08.209241 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:08.210031 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:08.210034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:08.210036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:08.210047 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:08.210073 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:08.210086 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:08.210159 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:08.210261 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:40:08.210294 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:40:08.211190 27133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53000b7e54304f879bbfaec0c6167910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34003 } }
I20260928 14:40:08.211516 27133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:08.211580 27133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53000b7e54304f879bbfaec0c6167910, State: Initialized, Role: FOLLOWER
I20260928 14:40:08.212041 27133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53000b7e54304f879bbfaec0c6167910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34003 } }
I20260928 14:40:08.212167 27133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:08.212214 27133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:08.212281 27133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:08.213168 27133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53000b7e54304f879bbfaec0c6167910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34003 } }
I20260928 14:40:08.213474 27133 leader_election.cc:302] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53000b7e54304f879bbfaec0c6167910; no voters:
I20260928 14:40:08.213726 27133 leader_election.cc:288] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:08.213850 27143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:08.214098 27143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [term 1 LEADER]: Becoming Leader. State: Replica: 53000b7e54304f879bbfaec0c6167910, State: Running, Role: LEADER
I20260928 14:40:08.214464 27143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53000b7e54304f879bbfaec0c6167910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34003 } }
I20260928 14:40:08.215020 27133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:08.216284 27145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53000b7e54304f879bbfaec0c6167910. Latest consensus state: current_term: 1 leader_uuid: "53000b7e54304f879bbfaec0c6167910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53000b7e54304f879bbfaec0c6167910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34003 } } }
I20260928 14:40:08.216436 27145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:08.216282 27144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53000b7e54304f879bbfaec0c6167910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53000b7e54304f879bbfaec0c6167910" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34003 } } }
I20260928 14:40:08.216670 27144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:08.217012 27152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:08.220268 27152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:08.225669 27152 catalog_manager.cc:1347] Generated new cluster ID: 9ac781ba6d114320a4ec1dad55349b6a
I20260928 14:40:08.225749 27152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:08.239806 27152 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:08.240170 27152 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:08.254281 27152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53000b7e54304f879bbfaec0c6167910: Generated new TSK 0
I20260928 14:40:08.255102 27152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:08.259097 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27065
2026-09-28T14:40:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:08 dist-test-slave-06lh krb5kdc[27164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:08 dist-test-slave-06lh krb5kdc[27164](info): set up 2 sockets
Sep 28 14:40:08 dist-test-slave-06lh krb5kdc[27164](info): commencing operation
krb5kdc: starting...
W20260928 14:40:09.319734 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:09 dist-test-slave-06lh krb5kdc[27164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:09.446240 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:09.560271 27177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:09.560634 27177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:09.560726 27177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:09.564926 27177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:09.565025 27177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:09.565063 27177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:09.565124 27177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:09.565162 27177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:09.570679 27177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:09.571919 27177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:09.572989 27177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:09.581876 27185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:09.581941 27177 server_base.cc:1034] running on GCE node
W20260928 14:40:09.581856 27183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:09.581856 27182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:09.582788 27177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:09.583997 27177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:09.585160 27177 hybrid_clock.cc:648] HybridClock initialized: now 1790606409585140 us; error 34 us; skew 500 ppm
Sep 28 14:40:09 dist-test-slave-06lh krb5kdc[27164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:09.587966 27177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:09.588889 27177 webserver.cc:460] Webserver started at http://127.0.0.1:36017/ using document root <none> and password file <none>
I20260928 14:40:09.589428 27177 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:09.589485 27177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:09.589651 27177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:09.591722 27177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "84987610fbe14e00b6f3196f07678ff0"
format_stamp: "Formatted at 2026-09-28 14:40:09 on dist-test-slave-06lh"
server_key: "2989fc5f085dddced295d7c66112cb94"
server_key_iv: "e865a6ad280135f38b7564a09e9a7bee"
server_key_version: "encryptionkey@0"
I20260928 14:40:09.592315 27177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "84987610fbe14e00b6f3196f07678ff0"
format_stamp: "Formatted at 2026-09-28 14:40:09 on dist-test-slave-06lh"
server_key: "2989fc5f085dddced295d7c66112cb94"
server_key_iv: "e865a6ad280135f38b7564a09e9a7bee"
server_key_version: "encryptionkey@0"
I20260928 14:40:09.596031 27177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:40:09.598400 27192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:09.599686 27177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:09.599833 27177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "84987610fbe14e00b6f3196f07678ff0"
format_stamp: "Formatted at 2026-09-28 14:40:09 on dist-test-slave-06lh"
server_key: "2989fc5f085dddced295d7c66112cb94"
server_key_iv: "e865a6ad280135f38b7564a09e9a7bee"
server_key_version: "encryptionkey@0"
I20260928 14:40:09.599951 27177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:09.629652 27177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:09.632772 27177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:09.633004 27177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:09.640416 27177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42895
I20260928 14:40:09.640436 27244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42895 every 8 connection(s)
I20260928 14:40:09.641426 27177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:09.643103 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27177
I20260928 14:40:09.643210 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:09.643554 6239 external_mini_cluster.cc:1428] Setting key 03a3d6752277f7e4f8bffdec4b38e1be
I20260928 14:40:09.644538 27245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:09.651556 27245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0: Bootstrap starting.
Sep 28 14:40:09 dist-test-slave-06lh krb5kdc[27164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:40:09.654387 27245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:09.655406 27245 log.cc:824] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:09.657931 27245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0: No bootstrap required, opened a new log
I20260928 14:40:09.659368 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:09.645049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:47697 (local address 127.0.0.1:42895)
0928 14:40:09.645486 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:09.645506 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:09.645550 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:40:09.646414 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:09.646439 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:09.646885 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:09.647321 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:09.648052 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:09.649568 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:09.650494 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:09.651170 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:09.651384 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:09.653899 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:09.653906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:09.653923 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:09.653960 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:09.656020 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:09.656603 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:09.656610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:09.656616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:09.656750 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:09.657074 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:09.657077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:09.657079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:09.657587 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:09.657800 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:09.658090 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 14:40:09.658341 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":130,"threads_started":1}
I20260928 14:40:09.662547 27245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84987610fbe14e00b6f3196f07678ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } }
I20260928 14:40:09.662853 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:09.662853 27245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:09.662928 27245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84987610fbe14e00b6f3196f07678ff0, State: Initialized, Role: FOLLOWER
I20260928 14:40:09.663450 27245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84987610fbe14e00b6f3196f07678ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } }
I20260928 14:40:09.663576 27245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:09.663662 27245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:09.663785 27245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:09.664923 27245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84987610fbe14e00b6f3196f07678ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } }
I20260928 14:40:09.665321 27245 leader_election.cc:302] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84987610fbe14e00b6f3196f07678ff0; no voters:
I20260928 14:40:09.665661 27245 leader_election.cc:288] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:09.665814 27255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:09.666100 27255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [term 1 LEADER]: Becoming Leader. State: Replica: 84987610fbe14e00b6f3196f07678ff0, State: Running, Role: LEADER
I20260928 14:40:09.666455 27255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84987610fbe14e00b6f3196f07678ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } }
I20260928 14:40:09.667090 27245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:09.667640 27255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84987610fbe14e00b6f3196f07678ff0. Latest consensus state: current_term: 1 leader_uuid: "84987610fbe14e00b6f3196f07678ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84987610fbe14e00b6f3196f07678ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } } }
I20260928 14:40:09.667840 27255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:09.668083 27257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84987610fbe14e00b6f3196f07678ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84987610fbe14e00b6f3196f07678ff0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42895 } } }
I20260928 14:40:09.668174 27257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:09.668289 27261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:09.671605 27261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:09.674816 27248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:09.665199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:36647 (local address 127.0.0.1:42895)
0928 14:40:09.665337 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:09.665341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:09.665521 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:40:09.665677 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:09.665683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:09.665738 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:09.665817 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:09.666633 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:09.667444 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:09.668668 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:09.669058 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:09.669166 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:09.670961 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:09.670964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:09.670967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:09.670993 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:09.672742 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:09.673355 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:09.673359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:09.673361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:09.673448 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:09.673839 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:09.673843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:09.673844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:09.674079 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:09.674232 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:09.674550 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 14:40:09.674655 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:40:09.677897 27261 catalog_manager.cc:1347] Generated new cluster ID: 80d85327d0f941e4bbbbf5133a35c0b8
I20260928 14:40:09.677983 27261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:09.704573 27261 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:09.705566 27261 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:09.710356 27261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84987610fbe14e00b6f3196f07678ff0: Generated new TSK 0
I20260928 14:40:09.711071 27261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:09.719904 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27177
W20260928 14:40:09.725894 27162 connection.cc:535] client connection to 127.0.0.1:42895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42895): Connection reset by peer (error 104)
2026-09-28T14:40:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1461 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:09 dist-test-slave-06lh krb5kdc[27276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:09 dist-test-slave-06lh krb5kdc[27276](info): set up 2 sockets
Sep 28 14:40:09 dist-test-slave-06lh krb5kdc[27276](info): commencing operation
krb5kdc: starting...
W20260928 14:40:10.771901 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:10 dist-test-slave-06lh krb5kdc[27276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:10.898957 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32841
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:11.014605 27289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:11.014933 27289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:11.015033 27289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:11.019332 27289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:11.019451 27289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:11.019488 27289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:11.019529 27289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:11.019564 27289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:11.025385 27289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:11.026597 27289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:11.027752 27289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:40:11.037031 27289 server_base.cc:1034] running on GCE node
W20260928 14:40:11.036978 27297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:11.036988 27295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:11.036985 27294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:11.037871 27289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:11.039021 27289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:11.040267 27289 hybrid_clock.cc:648] HybridClock initialized: now 1790606411040269 us; error 62 us; skew 500 ppm
Sep 28 14:40:11 dist-test-slave-06lh krb5kdc[27276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:11.050374 27289 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:11.051527 27289 webserver.cc:460] Webserver started at http://127.0.0.1:33845/ using document root <none> and password file <none>
I20260928 14:40:11.052166 27289 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:11.052268 27289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:11.052515 27289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:11.054759 27289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "7343f420b6fb4395a2a1179fe82eafe2"
format_stamp: "Formatted at 2026-09-28 14:40:11 on dist-test-slave-06lh"
server_key: "2c606de5ad28b99b8f13d7b628634f08"
server_key_iv: "410568a1b77b474f740e51b905f28a51"
server_key_version: "encryptionkey@0"
I20260928 14:40:11.055418 27289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "7343f420b6fb4395a2a1179fe82eafe2"
format_stamp: "Formatted at 2026-09-28 14:40:11 on dist-test-slave-06lh"
server_key: "2c606de5ad28b99b8f13d7b628634f08"
server_key_iv: "410568a1b77b474f740e51b905f28a51"
server_key_version: "encryptionkey@0"
I20260928 14:40:11.059522 27289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:40:11.062216 27304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:11.063601 27289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:40:11.063762 27289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "7343f420b6fb4395a2a1179fe82eafe2"
format_stamp: "Formatted at 2026-09-28 14:40:11 on dist-test-slave-06lh"
server_key: "2c606de5ad28b99b8f13d7b628634f08"
server_key_iv: "410568a1b77b474f740e51b905f28a51"
server_key_version: "encryptionkey@0"
I20260928 14:40:11.063894 27289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:11.074412 27289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:11.077864 27289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:11.078084 27289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:11.085457 27289 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32841
I20260928 14:40:11.085477 27356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32841 every 8 connection(s)
I20260928 14:40:11.086524 27289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:11.089438 27357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:11.095031 27357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2: Bootstrap starting.
I20260928 14:40:11.096053 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27289
I20260928 14:40:11.096207 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:11.096535 6239 external_mini_cluster.cc:1428] Setting key 064a47cf870293b1a539fd9c02496522
I20260928 14:40:11.098258 27357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:11.099010 27357 log.cc:824] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:11.101001 27357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2: No bootstrap required, opened a new log
I20260928 14:40:11.104971 27357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7343f420b6fb4395a2a1179fe82eafe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } }
I20260928 14:40:11.105309 27357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:11.105429 27357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7343f420b6fb4395a2a1179fe82eafe2, State: Initialized, Role: FOLLOWER
I20260928 14:40:11.105901 27357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7343f420b6fb4395a2a1179fe82eafe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } }
I20260928 14:40:11.106070 27357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:11.106150 27357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:11.106279 27357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:11.106341 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:11.097750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:35951 (local address 127.0.0.1:32841)
0928 14:40:11.098169 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:11.098197 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:11.098235 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:40:11.098956 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:11.098974 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:11.099449 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:11.099806 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:11.100374 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:11.101454 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:11.102400 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:11.103021 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:11.103268 (+ 247us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:11.104329 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:11.104336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:11.104350 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:11.104756 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:11.104812 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:11.104836 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:11.104978 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:11.105134 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:40:11.105288 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":136,"threads_started":1}
I20260928 14:40:11.107581 27357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7343f420b6fb4395a2a1179fe82eafe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } }
I20260928 14:40:11.107949 27357 leader_election.cc:302] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7343f420b6fb4395a2a1179fe82eafe2; no voters:
I20260928 14:40:11.108355 27357 leader_election.cc:288] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:11.108467 27362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:11.108776 27362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [term 1 LEADER]: Becoming Leader. State: Replica: 7343f420b6fb4395a2a1179fe82eafe2, State: Running, Role: LEADER
I20260928 14:40:11.109140 27362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7343f420b6fb4395a2a1179fe82eafe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } }
I20260928 14:40:11.109794 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:11.109817 27357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:11.111938 27364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7343f420b6fb4395a2a1179fe82eafe2. Latest consensus state: current_term: 1 leader_uuid: "7343f420b6fb4395a2a1179fe82eafe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7343f420b6fb4395a2a1179fe82eafe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } } }
I20260928 14:40:11.112077 27364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:11.112243 27363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7343f420b6fb4395a2a1179fe82eafe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7343f420b6fb4395a2a1179fe82eafe2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32841 } } }
I20260928 14:40:11.112840 27363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2 [sys.catalog]: This master's current role is: LEADER
W20260928 14:40:11.115110 27380 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:11.115183 27380 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:11.115299 27382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:11.116279 27382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:11.120026 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:11.112245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:55233 (local address 127.0.0.1:32841)
0928 14:40:11.112339 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:11.112344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:11.115601 (+ 3257us) server_negotiation.cc:407] Connection header received
0928 14:40:11.115764 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:11.115767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:11.115824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:11.115915 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:11.116268 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:11.117075 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:11.118226 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:11.118481 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:11.118598 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:11.119539 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:11.119542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:11.119545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:11.119560 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:11.119594 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:11.119605 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:11.119703 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:11.119828 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 14:40:11.119865 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 14:40:11.122625 27382 catalog_manager.cc:1347] Generated new cluster ID: cb87af1e553b4bed8dcb2cb6ebfa1f91
I20260928 14:40:11.122701 27382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:11.137759 27382 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:11.139135 27382 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:11.149945 27382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7343f420b6fb4395a2a1179fe82eafe2: Generated new TSK 0
I20260928 14:40:11.150817 27382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:11.161835 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27289
W20260928 14:40:11.167480 27274 connection.cc:535] client connection to 127.0.0.1:32841 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32841): Connection reset by peer (error 104)
2026-09-28T14:40:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:11 dist-test-slave-06lh krb5kdc[27388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:11 dist-test-slave-06lh krb5kdc[27388](info): set up 2 sockets
Sep 28 14:40:11 dist-test-slave-06lh krb5kdc[27388](info): commencing operation
krb5kdc: starting...
W20260928 14:40:12.199731 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:12 dist-test-slave-06lh krb5kdc[27388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:12.327494 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:12.441030 27401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:12.441366 27401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:12.441461 27401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:12.445636 27401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:12.445739 27401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:12.445777 27401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:12.445833 27401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:12.445886 27401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:12.451264 27401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:12.452411 27401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:12.453723 27401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:12.462384 27407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:12.462414 27409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:12.462384 27406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:12.462637 27401 server_base.cc:1034] running on GCE node
I20260928 14:40:12.463286 27401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:12.464375 27401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:12.465595 27401 hybrid_clock.cc:648] HybridClock initialized: now 1790606412465578 us; error 27 us; skew 500 ppm
Sep 28 14:40:12 dist-test-slave-06lh krb5kdc[27388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:12.468472 27401 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:12.469368 27401 webserver.cc:460] Webserver started at http://127.0.0.1:35263/ using document root <none> and password file <none>
I20260928 14:40:12.469936 27401 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:12.470024 27401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:12.470252 27401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:12.472433 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "6cb828710ce847ac8edab211a2da52eb"
format_stamp: "Formatted at 2026-09-28 14:40:12 on dist-test-slave-06lh"
server_key: "ef630f3cf71ad5a9def9bffbe20a395f"
server_key_iv: "eafc0f8e0e5462e2128350d1977880ea"
server_key_version: "encryptionkey@0"
I20260928 14:40:12.473008 27401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "6cb828710ce847ac8edab211a2da52eb"
format_stamp: "Formatted at 2026-09-28 14:40:12 on dist-test-slave-06lh"
server_key: "ef630f3cf71ad5a9def9bffbe20a395f"
server_key_iv: "eafc0f8e0e5462e2128350d1977880ea"
server_key_version: "encryptionkey@0"
I20260928 14:40:12.476873 27401 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:40:12.479401 27416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:12.480659 27401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:12.480805 27401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "6cb828710ce847ac8edab211a2da52eb"
format_stamp: "Formatted at 2026-09-28 14:40:12 on dist-test-slave-06lh"
server_key: "ef630f3cf71ad5a9def9bffbe20a395f"
server_key_iv: "eafc0f8e0e5462e2128350d1977880ea"
server_key_version: "encryptionkey@0"
I20260928 14:40:12.480922 27401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:12.485579 27401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:12.488497 27401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:12.488689 27401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:12.495637 27401 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38527
I20260928 14:40:12.495648 27468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38527 every 8 connection(s)
I20260928 14:40:12.496567 27401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:12.499326 27469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:12.504029 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27401
I20260928 14:40:12.504204 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:12.504526 6239 external_mini_cluster.cc:1428] Setting key c5492516dd30ff83f4d395d1c8201375
I20260928 14:40:12.505086 27469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb: Bootstrap starting.
I20260928 14:40:12.507680 27469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:12.508558 27469 log.cc:824] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:12.509897 27472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:12.505973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:47721 (local address 127.0.0.1:38527)
0928 14:40:12.506427 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:12.506441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:12.506477 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:40:12.507147 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:12.507164 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:12.507526 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:12.507871 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:12.508684 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:12.508691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:12.508709 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:12.508724 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:12.508792 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:12.508814 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:12.508932 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:12.509036 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 14:40:12.509205 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260928 14:40:12.510658 27469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb: No bootstrap required, opened a new log
I20260928 14:40:12.512758 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:12.514683 27469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb828710ce847ac8edab211a2da52eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } }
I20260928 14:40:12.514985 27469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:12.515056 27469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb828710ce847ac8edab211a2da52eb, State: Initialized, Role: FOLLOWER
I20260928 14:40:12.515580 27469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb828710ce847ac8edab211a2da52eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } }
I20260928 14:40:12.515787 27469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:12.515879 27469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:12.515990 27469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:12.516583 27472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:12.514747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:53051 (local address 127.0.0.1:38527)
0928 14:40:12.514868 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:12.514872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:12.515095 (+ 223us) server_negotiation.cc:407] Connection header received
0928 14:40:12.515275 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:12.515278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:12.515320 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:12.515405 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:12.516186 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:12.516190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:12.516192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:12.516203 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:12.516227 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:12.516237 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:12.516308 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:12.516417 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 14:40:12.516449 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:40:12.517176 27469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb828710ce847ac8edab211a2da52eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } }
I20260928 14:40:12.517547 27469 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb828710ce847ac8edab211a2da52eb; no voters:
I20260928 14:40:12.517879 27469 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:12.517993 27479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:12.518226 27479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [term 1 LEADER]: Becoming Leader. State: Replica: 6cb828710ce847ac8edab211a2da52eb, State: Running, Role: LEADER
I20260928 14:40:12.518615 27479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb828710ce847ac8edab211a2da52eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } }
I20260928 14:40:12.519081 27469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:12.520342 27480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb828710ce847ac8edab211a2da52eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb828710ce847ac8edab211a2da52eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } } }
I20260928 14:40:12.520342 27481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb828710ce847ac8edab211a2da52eb. Latest consensus state: current_term: 1 leader_uuid: "6cb828710ce847ac8edab211a2da52eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb828710ce847ac8edab211a2da52eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38527 } } }
I20260928 14:40:12.520521 27480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:12.520534 27481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb [sys.catalog]: This master's current role is: LEADER
W20260928 14:40:12.524163 27494 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:12.524245 27494 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:12.524329 27490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:12.525039 27490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:12.530342 27490 catalog_manager.cc:1347] Generated new cluster ID: 989d8b4aa5c9408db89a6a84e18e39c4
I20260928 14:40:12.530416 27490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:12.557276 27490 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:12.557657 27490 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:12.567970 27490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb828710ce847ac8edab211a2da52eb: Generated new TSK 0
I20260928 14:40:12.568708 27490 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:12.575102 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27401
2026-09-28T14:40:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:12 dist-test-slave-06lh krb5kdc[27500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:12 dist-test-slave-06lh krb5kdc[27500](info): set up 2 sockets
Sep 28 14:40:12 dist-test-slave-06lh krb5kdc[27500](info): commencing operation
krb5kdc: starting...
W20260928 14:40:13.615913 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:13 dist-test-slave-06lh krb5kdc[27500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:13.738689 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44389
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:13.849460 27513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:13.849738 27513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:13.849781 27513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:13.853725 27513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:13.853832 27513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:13.853873 27513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:13.853904 27513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:13.853926 27513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:13.859310 27513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:13.860282 27513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:13.861227 27513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:13.870362 27521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:13.870501 27518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:13.870407 27519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:13.870397 27513 server_base.cc:1034] running on GCE node
I20260928 14:40:13.871172 27513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:13.872314 27513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:13.873499 27513 hybrid_clock.cc:648] HybridClock initialized: now 1790606413873485 us; error 26 us; skew 500 ppm
Sep 28 14:40:13 dist-test-slave-06lh krb5kdc[27500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:13.876358 27513 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:13.877230 27513 webserver.cc:460] Webserver started at http://127.0.0.1:39295/ using document root <none> and password file <none>
I20260928 14:40:13.877794 27513 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:13.877884 27513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:13.878120 27513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:13.880226 27513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5"
format_stamp: "Formatted at 2026-09-28 14:40:13 on dist-test-slave-06lh"
server_key: "f5b041daf3eaca0f01c62e10912acbbc"
server_key_iv: "c4dcb519d4fd2fd5297b1d0200aa1e5e"
server_key_version: "encryptionkey@0"
I20260928 14:40:13.880790 27513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5"
format_stamp: "Formatted at 2026-09-28 14:40:13 on dist-test-slave-06lh"
server_key: "f5b041daf3eaca0f01c62e10912acbbc"
server_key_iv: "c4dcb519d4fd2fd5297b1d0200aa1e5e"
server_key_version: "encryptionkey@0"
I20260928 14:40:13.884377 27513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 14:40:13.886703 27528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:13.887962 27513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 14:40:13.888108 27513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5"
format_stamp: "Formatted at 2026-09-28 14:40:13 on dist-test-slave-06lh"
server_key: "f5b041daf3eaca0f01c62e10912acbbc"
server_key_iv: "c4dcb519d4fd2fd5297b1d0200aa1e5e"
server_key_version: "encryptionkey@0"
I20260928 14:40:13.888221 27513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:13.909334 27513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:13.912643 27513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:13.912860 27513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:13.919864 27513 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44389
I20260928 14:40:13.919883 27580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44389 every 8 connection(s)
I20260928 14:40:13.920848 27513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:13.923593 27581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:13.925153 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27513
I20260928 14:40:13.925310 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:13.925630 6239 external_mini_cluster.cc:1428] Setting key df9a6bf0d9c0e0252bec043abb00e196
I20260928 14:40:13.929930 27581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5: Bootstrap starting.
I20260928 14:40:13.932291 27581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:13.933017 27581 log.cc:824] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:40:13 dist-test-slave-06lh krb5kdc[27500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:40:13.935371 27581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5: No bootstrap required, opened a new log
I20260928 14:40:13.938941 27581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } }
I20260928 14:40:13.939276 27581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:13.939360 27581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c9c3a2dea54ddfb1125b7e32e55ed5, State: Initialized, Role: FOLLOWER
I20260928 14:40:13.939832 27581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } }
I20260928 14:40:13.940011 27581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:13.940094 27581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:13.940220 27581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:13.941216 27581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } }
I20260928 14:40:13.941335 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:13.927259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:54419 (local address 127.0.0.1:44389)
0928 14:40:13.927736 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:13.927752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:13.927792 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:40:13.928429 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:13.928446 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:13.928803 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:13.929161 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:13.930025 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:13.931042 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:13.931996 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:13.932533 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:13.932696 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:13.935718 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:13.935723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:13.935736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:13.935762 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:13.937957 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:13.938593 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:13.938603 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:13.938611 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:13.938747 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:13.939117 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:13.939127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:13.939128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:13.939706 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:13.939898 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:13.940319 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 14:40:13.940539 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":143,"threads_started":1}
I20260928 14:40:13.941596 27581 leader_election.cc:302] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c9c3a2dea54ddfb1125b7e32e55ed5; no voters:
I20260928 14:40:13.941954 27581 leader_election.cc:288] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:13.942061 27586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:13.942224 27586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [term 1 LEADER]: Becoming Leader. State: Replica: 39c9c3a2dea54ddfb1125b7e32e55ed5, State: Running, Role: LEADER
I20260928 14:40:13.943298 27581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:13.943435 27586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } }
I20260928 14:40:13.944628 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:13.945663 27588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c9c3a2dea54ddfb1125b7e32e55ed5. Latest consensus state: current_term: 1 leader_uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } } }
I20260928 14:40:13.946919 27588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:13.945842 27587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c3a2dea54ddfb1125b7e32e55ed5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44389 } } }
I20260928 14:40:13.947180 27587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5 [sys.catalog]: This master's current role is: LEADER
W20260928 14:40:13.948243 27600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:13.948333 27600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:13.950457 27606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:13.951476 27606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:13.956358 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:13.947271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:37011 (local address 127.0.0.1:44389)
0928 14:40:13.947395 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:13.947401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:13.947609 (+ 208us) server_negotiation.cc:407] Connection header received
0928 14:40:13.947726 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:13.947729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:13.947778 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:13.947870 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:13.948727 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:13.949471 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:13.950659 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:13.950958 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:13.951074 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:13.952847 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:13.952852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:13.952855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:13.952889 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:13.954539 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:13.955027 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:13.955030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:13.955032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:13.955110 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:13.955469 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:13.955472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:13.955474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:13.955658 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:13.955810 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:13.956083 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 14:40:13.956177 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:40:13.958333 27606 catalog_manager.cc:1347] Generated new cluster ID: 0d834f596576479d8ae0d7cbf4600bd9
I20260928 14:40:13.958395 27606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:13.985056 27606 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:13.986328 27606 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:13.998878 27606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39c9c3a2dea54ddfb1125b7e32e55ed5: Generated new TSK 0
I20260928 14:40:13.999861 27606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:14.012375 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27513
W20260928 14:40:14.017895 27498 connection.cc:535] client connection to 127.0.0.1:44389 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44389): Connection reset by peer (error 104)
2026-09-28T14:40:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1437 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:14 dist-test-slave-06lh krb5kdc[27612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:14 dist-test-slave-06lh krb5kdc[27612](info): set up 2 sockets
Sep 28 14:40:14 dist-test-slave-06lh krb5kdc[27612](info): commencing operation
krb5kdc: starting...
W20260928 14:40:15.043826 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:15 dist-test-slave-06lh krb5kdc[27612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:15.166440 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39327
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:15.277321 27625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:15.277634 27625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:15.277714 27625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:15.281739 27625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:15.281832 27625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:15.281872 27625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:15.281932 27625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:15.281970 27625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:15.287185 27625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39327
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:15.288297 27625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:15.289376 27625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:15.297819 27633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:15.297838 27630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:15.298002 27625 server_base.cc:1034] running on GCE node
W20260928 14:40:15.297870 27631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:15.298681 27625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:15.299835 27625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:15.301043 27625 hybrid_clock.cc:648] HybridClock initialized: now 1790606415301010 us; error 44 us; skew 500 ppm
Sep 28 14:40:15 dist-test-slave-06lh krb5kdc[27612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:15.303879 27625 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:15.304777 27625 webserver.cc:460] Webserver started at http://127.0.0.1:33089/ using document root <none> and password file <none>
I20260928 14:40:15.305354 27625 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:15.305442 27625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:15.305668 27625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:15.307783 27625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "a7aef94c2c954b6fb868cd597363c58b"
format_stamp: "Formatted at 2026-09-28 14:40:15 on dist-test-slave-06lh"
server_key: "8095bc59254ebedf547913213ac7f9ad"
server_key_iv: "6d70eace2713535728de0f1dbf00773f"
server_key_version: "encryptionkey@0"
I20260928 14:40:15.308343 27625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "a7aef94c2c954b6fb868cd597363c58b"
format_stamp: "Formatted at 2026-09-28 14:40:15 on dist-test-slave-06lh"
server_key: "8095bc59254ebedf547913213ac7f9ad"
server_key_iv: "6d70eace2713535728de0f1dbf00773f"
server_key_version: "encryptionkey@0"
I20260928 14:40:15.312067 27625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:40:15.314491 27640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:15.315776 27625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:15.315910 27625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "a7aef94c2c954b6fb868cd597363c58b"
format_stamp: "Formatted at 2026-09-28 14:40:15 on dist-test-slave-06lh"
server_key: "8095bc59254ebedf547913213ac7f9ad"
server_key_iv: "6d70eace2713535728de0f1dbf00773f"
server_key_version: "encryptionkey@0"
I20260928 14:40:15.316022 27625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:15.330483 27625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:15.333698 27625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:15.333850 27625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:15.340710 27625 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35717
I20260928 14:40:15.340730 27692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35717 every 8 connection(s)
I20260928 14:40:15.341717 27625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:15.342519 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27625
I20260928 14:40:15.342662 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:15.342947 6239 external_mini_cluster.cc:1428] Setting key aabf96730f6494f57e53390b10edd387
I20260928 14:40:15.344847 27693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:15.351326 27693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b: Bootstrap starting.
I20260928 14:40:15.352159 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:15.344414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:47823 (local address 127.0.0.1:35717)
0928 14:40:15.344813 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:15.344837 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:15.344872 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:40:15.345548 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:15.345565 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:15.345903 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:15.346192 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:15.346569 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:15.347664 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:15.348901 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:15.349360 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:15.349470 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:15.350437 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:15.350442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:15.350454 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:15.350850 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:15.350904 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:15.350926 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:15.351026 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:15.351311 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 14:40:15.351457 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260928 14:40:15.353844 27693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:15.354689 27693 log.cc:824] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:15.355206 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:15.356714 27693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b: No bootstrap required, opened a new log
I20260928 14:40:15.360726 27693 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aef94c2c954b6fb868cd597363c58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260928 14:40:15.361037 27693 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:15.361107 27693 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7aef94c2c954b6fb868cd597363c58b, State: Initialized, Role: FOLLOWER
I20260928 14:40:15.361563 27693 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aef94c2c954b6fb868cd597363c58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260928 14:40:15.361712 27693 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:15.361806 27693 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:15.361930 27693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:15.363143 27693 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aef94c2c954b6fb868cd597363c58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260928 14:40:15.363548 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:15.357228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:43975 (local address 127.0.0.1:35717)
0928 14:40:15.357320 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:15.357323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:15.359533 (+ 2210us) server_negotiation.cc:407] Connection header received
0928 14:40:15.359730 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:15.359733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:15.359790 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:15.359884 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:15.360208 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:15.360708 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:15.361821 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:15.362140 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:15.362228 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:15.363066 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:15.363069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:15.363070 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:15.363081 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:15.363103 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:15.363112 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:15.363199 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:15.363418 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 14:40:15.363450 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 14:40:15.363564 27693 leader_election.cc:302] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7aef94c2c954b6fb868cd597363c58b; no voters:
I20260928 14:40:15.363919 27693 leader_election.cc:288] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:15.364010 27703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:15.364238 27703 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [term 1 LEADER]: Becoming Leader. State: Replica: a7aef94c2c954b6fb868cd597363c58b, State: Running, Role: LEADER
I20260928 14:40:15.364559 27703 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aef94c2c954b6fb868cd597363c58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } }
I20260928 14:40:15.365222 27693 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:15.366173 27705 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7aef94c2c954b6fb868cd597363c58b. Latest consensus state: current_term: 1 leader_uuid: "a7aef94c2c954b6fb868cd597363c58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aef94c2c954b6fb868cd597363c58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } } }
I20260928 14:40:15.366328 27705 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:15.366729 27712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:15.366709 27704 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7aef94c2c954b6fb868cd597363c58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7aef94c2c954b6fb868cd597363c58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35717 } } }
I20260928 14:40:15.367313 27704 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:15.369736 27712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:15.374852 27712 catalog_manager.cc:1347] Generated new cluster ID: 397b8801e40944afa8f449c4060d213f
I20260928 14:40:15.374917 27712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:15.386981 27712 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:15.387907 27712 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:15.392933 27712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7aef94c2c954b6fb868cd597363c58b: Generated new TSK 0
I20260928 14:40:15.393623 27712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:15.399053 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27625
W20260928 14:40:15.404728 27610 connection.cc:535] client connection to 127.0.0.1:35717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35717): Connection reset by peer (error 104)
2026-09-28T14:40:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:15 dist-test-slave-06lh krb5kdc[27724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:15 dist-test-slave-06lh krb5kdc[27724](info): set up 2 sockets
Sep 28 14:40:15 dist-test-slave-06lh krb5kdc[27724](info): commencing operation
krb5kdc: starting...
W20260928 14:40:16.431751 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:16 dist-test-slave-06lh krb5kdc[27724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:40:16.554953 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:16.668828 27737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:16.669312 27737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:16.669459 27737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:16.673728 27737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:16.673799 27737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:16.673820 27737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:16.673838 27737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:16.673885 27737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:16.679704 27737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42715
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:16.681387 27737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:16.682948 27737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:16.692675 27742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:16.692678 27745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:16.692811 27737 server_base.cc:1034] running on GCE node
W20260928 14:40:16.692664 27743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:16.693535 27737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:16.694599 27737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:16.695761 27737 hybrid_clock.cc:648] HybridClock initialized: now 1790606416695747 us; error 25 us; skew 500 ppm
Sep 28 14:40:16 dist-test-slave-06lh krb5kdc[27724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:16.702229 27737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:40:16.703135 27737 webserver.cc:460] Webserver started at http://127.0.0.1:46841/ using document root <none> and password file <none>
I20260928 14:40:16.703733 27737 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:16.703796 27737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:16.703962 27737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:16.705984 27737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "2e3db1915fed4b3f85e11f3e4ef7870a"
format_stamp: "Formatted at 2026-09-28 14:40:16 on dist-test-slave-06lh"
server_key: "a0bee5fd7b65e43b542ae55f5611a939"
server_key_iv: "114350403718733fdb9ed46394e06797"
server_key_version: "encryptionkey@0"
I20260928 14:40:16.706509 27737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "2e3db1915fed4b3f85e11f3e4ef7870a"
format_stamp: "Formatted at 2026-09-28 14:40:16 on dist-test-slave-06lh"
server_key: "a0bee5fd7b65e43b542ae55f5611a939"
server_key_iv: "114350403718733fdb9ed46394e06797"
server_key_version: "encryptionkey@0"
I20260928 14:40:16.710199 27737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 14:40:16.712688 27752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:16.713944 27737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:16.714089 27737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "2e3db1915fed4b3f85e11f3e4ef7870a"
format_stamp: "Formatted at 2026-09-28 14:40:16 on dist-test-slave-06lh"
server_key: "a0bee5fd7b65e43b542ae55f5611a939"
server_key_iv: "114350403718733fdb9ed46394e06797"
server_key_version: "encryptionkey@0"
I20260928 14:40:16.714210 27737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:16.718940 27737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:16.721952 27737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:16.722149 27737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:16.729394 27737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42715
I20260928 14:40:16.729413 27804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42715 every 8 connection(s)
I20260928 14:40:16.730366 27737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:16.731801 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27737
I20260928 14:40:16.731910 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:16.732247 6239 external_mini_cluster.cc:1428] Setting key 8a94cfd7514fce117e00cf757c3b8313
I20260928 14:40:16.733520 27805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:16.738046 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:16.733770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:42379 (local address 127.0.0.1:42715)
0928 14:40:16.734182 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:16.734201 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:16.734243 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:40:16.735102 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:16.735126 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:16.735511 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:16.735890 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:16.736642 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:16.736648 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:16.736665 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:16.736678 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:16.736743 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:16.736770 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:16.736914 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:16.736982 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:40:16.737139 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260928 14:40:16.740319 27805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a: Bootstrap starting.
I20260928 14:40:16.740854 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:16.742786 27805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:16.743752 27805 log.cc:824] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:16.744750 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:16.742854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.37:43475 (local address 127.0.0.1:42715)
0928 14:40:16.742963 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:16.742970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:16.743268 (+ 298us) server_negotiation.cc:407] Connection header received
0928 14:40:16.743410 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:16.743413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:16.743467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:16.743553 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:16.744328 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:16.744331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:16.744334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:16.744348 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:16.744377 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:16.744387 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:16.744482 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:16.744591 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 14:40:16.744637 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 14:40:16.745951 27805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a: No bootstrap required, opened a new log
I20260928 14:40:16.749733 27805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3db1915fed4b3f85e11f3e4ef7870a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } }
I20260928 14:40:16.749993 27805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:16.750032 27805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3db1915fed4b3f85e11f3e4ef7870a, State: Initialized, Role: FOLLOWER
I20260928 14:40:16.750411 27805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3db1915fed4b3f85e11f3e4ef7870a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } }
I20260928 14:40:16.750516 27805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:16.750555 27805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:16.750628 27805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:16.751571 27805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3db1915fed4b3f85e11f3e4ef7870a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } }
I20260928 14:40:16.751869 27805 leader_election.cc:302] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e3db1915fed4b3f85e11f3e4ef7870a; no voters:
I20260928 14:40:16.752127 27805 leader_election.cc:288] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:16.752259 27815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:16.752516 27815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [term 1 LEADER]: Becoming Leader. State: Replica: 2e3db1915fed4b3f85e11f3e4ef7870a, State: Running, Role: LEADER
I20260928 14:40:16.752848 27815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3db1915fed4b3f85e11f3e4ef7870a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } }
I20260928 14:40:16.753378 27805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:16.754634 27816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e3db1915fed4b3f85e11f3e4ef7870a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3db1915fed4b3f85e11f3e4ef7870a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } } }
I20260928 14:40:16.754781 27816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:16.754634 27817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e3db1915fed4b3f85e11f3e4ef7870a. Latest consensus state: current_term: 1 leader_uuid: "2e3db1915fed4b3f85e11f3e4ef7870a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e3db1915fed4b3f85e11f3e4ef7870a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42715 } } }
I20260928 14:40:16.754860 27817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:16.755283 27825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:16.758008 27825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:16.763882 27825 catalog_manager.cc:1347] Generated new cluster ID: e3d7a887882540d699d617a835e94ae6
I20260928 14:40:16.763964 27825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:16.782997 27825 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:16.783407 27825 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:16.793928 27825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e3db1915fed4b3f85e11f3e4ef7870a: Generated new TSK 0
I20260928 14:40:16.794652 27825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:16.801990 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27737
2026-09-28T14:40:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:16 dist-test-slave-06lh krb5kdc[27836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:16 dist-test-slave-06lh krb5kdc[27836](info): set up 2 sockets
Sep 28 14:40:16 dist-test-slave-06lh krb5kdc[27836](info): commencing operation
krb5kdc: starting...
W20260928 14:40:17.835842 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:17 dist-test-slave-06lh krb5kdc[27836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:40:17.958413 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40905
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:18.073567 27849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:18.073906 27849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:18.073995 27849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:18.078717 27849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:18.078804 27849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:18.078823 27849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:18.078841 27849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:18.078856 27849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:18.084481 27849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:40905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:18.085551 27849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:18.086552 27849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:18.095422 27855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:18.095404 27857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:18.095404 27854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:18.095872 27849 server_base.cc:1034] running on GCE node
I20260928 14:40:18.096529 27849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:18.097707 27849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:18.098917 27849 hybrid_clock.cc:648] HybridClock initialized: now 1790606418098908 us; error 37 us; skew 500 ppm
Sep 28 14:40:18 dist-test-slave-06lh krb5kdc[27836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:18.101801 27849 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:40:18.102778 27849 webserver.cc:460] Webserver started at http://127.6.23.254:38207/ using document root <none> and password file <none>
I20260928 14:40:18.103443 27849 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:18.103539 27849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:18.103777 27849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:18.106010 27849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "b4d75487486c40828572ff9d2ffc2028"
format_stamp: "Formatted at 2026-09-28 14:40:18 on dist-test-slave-06lh"
server_key: "ad66b0c0a7a461d6597c37a6f5d47d63"
server_key_iv: "b94b5164d3b5e7e25c3ce16461e94988"
server_key_version: "encryptionkey@0"
I20260928 14:40:18.106604 27849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "b4d75487486c40828572ff9d2ffc2028"
format_stamp: "Formatted at 2026-09-28 14:40:18 on dist-test-slave-06lh"
server_key: "ad66b0c0a7a461d6597c37a6f5d47d63"
server_key_iv: "b94b5164d3b5e7e25c3ce16461e94988"
server_key_version: "encryptionkey@0"
I20260928 14:40:18.110584 27849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 14:40:18.113236 27864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:18.114502 27849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:40:18.114670 27849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "b4d75487486c40828572ff9d2ffc2028"
format_stamp: "Formatted at 2026-09-28 14:40:18 on dist-test-slave-06lh"
server_key: "ad66b0c0a7a461d6597c37a6f5d47d63"
server_key_iv: "b94b5164d3b5e7e25c3ce16461e94988"
server_key_version: "encryptionkey@0"
I20260928 14:40:18.114776 27849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:18.139322 27849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:18.142884 27849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:18.143061 27849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:18.151124 27849 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:40905
I20260928 14:40:18.151153 27916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:40905 every 8 connection(s)
I20260928 14:40:18.152225 27849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:18.155324 27917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:18.155532 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27849
I20260928 14:40:18.155691 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:18.155993 6239 external_mini_cluster.cc:1428] Setting key 874c9aea8d8e4bfc73561d8cdffe5749
I20260928 14:40:18.162894 27917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028: Bootstrap starting.
I20260928 14:40:18.165544 27917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028: Neither blocks nor log segments found. Creating new log.
Sep 28 14:40:18 dist-test-slave-06lh krb5kdc[27836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:18.166539 27917 log.cc:824] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:18.169240 27917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028: No bootstrap required, opened a new log
I20260928 14:40:18.172596 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:18.157520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52895 (local address 127.6.23.254:40905)
0928 14:40:18.157990 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:18.158010 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:18.158055 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:40:18.158874 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:18.158900 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:18.159398 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:18.159804 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:18.160498 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:18.162036 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:18.163021 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:18.163753 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:18.163913 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:18.166757 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:18.166764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:18.166783 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:18.166816 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:18.168986 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:18.169689 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:18.169695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:18.169701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:18.169852 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:18.170169 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:18.170172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:18.170174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:18.170689 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:18.170903 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:18.171207 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 14:40:18.171482 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":139,"threads_started":1}
I20260928 14:40:18.173947 27917 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d75487486c40828572ff9d2ffc2028" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40905 } }
I20260928 14:40:18.174224 27917 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:18.174290 27917 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d75487486c40828572ff9d2ffc2028, State: Initialized, Role: FOLLOWER
I20260928 14:40:18.174786 27917 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d75487486c40828572ff9d2ffc2028" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40905 } }
I20260928 14:40:18.174935 27917 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:18.175022 27917 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:18.175124 27917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:18.176004 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:18.176422 27917 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d75487486c40828572ff9d2ffc2028" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40905 } }
I20260928 14:40:18.176787 27917 leader_election.cc:302] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4d75487486c40828572ff9d2ffc2028; no voters:
I20260928 14:40:18.177147 27917 leader_election.cc:288] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:18.177223 27922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:18.177435 27922 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [term 1 LEADER]: Becoming Leader. State: Replica: b4d75487486c40828572ff9d2ffc2028, State: Running, Role: LEADER
I20260928 14:40:18.177753 27922 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d75487486c40828572ff9d2ffc2028" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40905 } }
I20260928 14:40:18.178527 27917 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:18.179591 27926 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4d75487486c40828572ff9d2ffc2028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d75487486c40828572ff9d2ffc2028" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40905 } } }
I20260928 14:40:18.179608 27927 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4d75487486c40828572ff9d2ffc2028. Latest consensus state: current_term: 1 leader_uuid: "b4d75487486c40828572ff9d2ffc2028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d75487486c40828572ff9d2ffc2028" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 40905 } } }
I20260928 14:40:18.179768 27926 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:18.179772 27927 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:18.180215 27936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:18.183593 27936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:18.188201 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:18.178233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33747 (local address 127.6.23.254:40905)
0928 14:40:18.178351 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:18.178355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:18.178760 (+ 405us) server_negotiation.cc:407] Connection header received
0928 14:40:18.178914 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:18.178917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:18.178969 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:18.179027 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:18.180900 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:18.181421 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:18.182582 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:18.182961 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:18.183035 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:18.184865 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:18.184868 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:18.184870 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:18.184891 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:18.186347 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:18.186832 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:18.186838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:18.186845 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:18.186919 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:18.187295 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:18.187300 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:18.187306 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:18.187475 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:18.187632 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:18.187913 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 14:40:18.188005 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:40:18.190204 27936 catalog_manager.cc:1347] Generated new cluster ID: 04bb1fcc0d2643159bb14224e9256ca2
I20260928 14:40:18.190271 27936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:18.207141 27936 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:18.208498 27936 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:18.217993 27936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4d75487486c40828572ff9d2ffc2028: Generated new TSK 0
I20260928 14:40:18.218958 27936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:18.229888 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27849
W20260928 14:40:18.235566 27834 connection.cc:535] client connection to 127.6.23.254:40905 recv error: Network error: failed to read from TLS socket (remote: 127.6.23.254:40905): Connection reset by peer (error 104)
2026-09-28T14:40:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:18 dist-test-slave-06lh krb5kdc[27948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:18 dist-test-slave-06lh krb5kdc[27948](info): set up 2 sockets
Sep 28 14:40:18 dist-test-slave-06lh krb5kdc[27948](info): commencing operation
krb5kdc: starting...
W20260928 14:40:19.291780 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:19 dist-test-slave-06lh krb5kdc[27948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:40:19.414737 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42501
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:42501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:19.526588 27961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:19.526933 27961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:19.527011 27961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:19.531172 27961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:19.531291 27961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:19.531361 27961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:19.531400 27961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:19.531437 27961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:19.536808 27961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:42501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:19.537945 27961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:19.539017 27961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:40:19.547784 27961 server_base.cc:1034] running on GCE node
W20260928 14:40:19.547770 27966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:19.547775 27967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:19.547765 27969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:19.548746 27961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:19.549885 27961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:19.551106 27961 hybrid_clock.cc:648] HybridClock initialized: now 1790606419551073 us; error 48 us; skew 500 ppm
Sep 28 14:40:19 dist-test-slave-06lh krb5kdc[27948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606419, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:19.554101 27961 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:40:19.555094 27961 webserver.cc:460] Webserver started at http://127.6.23.254:42475/ using document root <none> and password file <none>
I20260928 14:40:19.555747 27961 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:19.555846 27961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:19.556083 27961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:19.558156 27961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "b2ea05ee960c4cf4a33835b2f78927ac"
format_stamp: "Formatted at 2026-09-28 14:40:19 on dist-test-slave-06lh"
server_key: "3557441d3f38aa6e0673c3e3bb799abe"
server_key_iv: "e9c1cf83fba6a5b4e95639e6f4cd799f"
server_key_version: "encryptionkey@0"
I20260928 14:40:19.558755 27961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "b2ea05ee960c4cf4a33835b2f78927ac"
format_stamp: "Formatted at 2026-09-28 14:40:19 on dist-test-slave-06lh"
server_key: "3557441d3f38aa6e0673c3e3bb799abe"
server_key_iv: "e9c1cf83fba6a5b4e95639e6f4cd799f"
server_key_version: "encryptionkey@0"
I20260928 14:40:19.562546 27961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:40:19.564988 27976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:19.566202 27961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 14:40:19.566365 27961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "b2ea05ee960c4cf4a33835b2f78927ac"
format_stamp: "Formatted at 2026-09-28 14:40:19 on dist-test-slave-06lh"
server_key: "3557441d3f38aa6e0673c3e3bb799abe"
server_key_iv: "e9c1cf83fba6a5b4e95639e6f4cd799f"
server_key_version: "encryptionkey@0"
I20260928 14:40:19.566483 27961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:19.579464 27961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:19.582517 27961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:19.582748 27961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:19.589736 27961 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:42501
I20260928 14:40:19.589762 28028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:42501 every 8 connection(s)
I20260928 14:40:19.590689 27961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:19.591060 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 27961
I20260928 14:40:19.591246 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:19.591533 6239 external_mini_cluster.cc:1428] Setting key 1f7d6e37151280442c59e9c99153b094
I20260928 14:40:19.594076 28029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:19.601219 28029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac: Bootstrap starting.
I20260928 14:40:19.601986 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:19.592960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59025 (local address 127.6.23.254:42501)
0928 14:40:19.593397 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:19.593417 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:19.593463 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:40:19.594352 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:19.594376 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:19.594810 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:19.595196 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:19.595645 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:19.597189 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:19.598120 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:19.598816 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:19.598985 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:19.599906 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:19.599913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:19.599930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:19.600462 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:19.600532 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:19.600561 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:19.600701 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:19.600916 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 14:40:19.601051 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260928 14:40:19.603924 28029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:19.604873 28029 log.cc:824] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:19.605173 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:19.606879 28029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac: No bootstrap required, opened a new log
I20260928 14:40:19.611635 28032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:19.607332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33483 (local address 127.6.23.254:42501)
0928 14:40:19.607461 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:19.607465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:19.607668 (+ 203us) server_negotiation.cc:407] Connection header received
0928 14:40:19.607791 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:19.607794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:19.607830 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:19.607897 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:19.608193 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:19.608707 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:19.609906 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:19.610118 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:19.610195 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:19.611094 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:19.611097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:19.611099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:19.611110 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:19.611133 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:19.611143 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:19.611217 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:19.611413 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 14:40:19.611473 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:40:19.611653 28029 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ea05ee960c4cf4a33835b2f78927ac" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42501 } }
I20260928 14:40:19.612001 28029 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:19.612064 28029 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ea05ee960c4cf4a33835b2f78927ac, State: Initialized, Role: FOLLOWER
I20260928 14:40:19.612545 28029 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ea05ee960c4cf4a33835b2f78927ac" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42501 } }
I20260928 14:40:19.612718 28029 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:19.612795 28029 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:19.612885 28029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:19.613770 28029 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ea05ee960c4cf4a33835b2f78927ac" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42501 } }
I20260928 14:40:19.614154 28029 leader_election.cc:302] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2ea05ee960c4cf4a33835b2f78927ac; no voters:
I20260928 14:40:19.614530 28029 leader_election.cc:288] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:19.614645 28039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:19.614884 28039 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [term 1 LEADER]: Becoming Leader. State: Replica: b2ea05ee960c4cf4a33835b2f78927ac, State: Running, Role: LEADER
I20260928 14:40:19.615284 28039 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ea05ee960c4cf4a33835b2f78927ac" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42501 } }
I20260928 14:40:19.615893 28029 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:19.616971 28040 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2ea05ee960c4cf4a33835b2f78927ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ea05ee960c4cf4a33835b2f78927ac" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42501 } } }
I20260928 14:40:19.617115 28040 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:19.617393 28041 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ea05ee960c4cf4a33835b2f78927ac. Latest consensus state: current_term: 1 leader_uuid: "b2ea05ee960c4cf4a33835b2f78927ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ea05ee960c4cf4a33835b2f78927ac" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42501 } } }
I20260928 14:40:19.617486 28041 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:19.617511 28048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:19.620716 28048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:19.626757 28048 catalog_manager.cc:1347] Generated new cluster ID: 5d1995696f6f423c916cedd457bc2ac4
I20260928 14:40:19.626847 28048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:19.631490 28048 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:19.632681 28048 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:19.638547 28048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2ea05ee960c4cf4a33835b2f78927ac: Generated new TSK 0
I20260928 14:40:19.639281 28048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:19.646230 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 27961
W20260928 14:40:19.651896 27946 connection.cc:535] client connection to 127.6.23.254:42501 recv error: Network error: failed to read from TLS socket (remote: 127.6.23.254:42501): Connection reset by peer (error 104)
2026-09-28T14:40:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:19 dist-test-slave-06lh krb5kdc[28060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:19 dist-test-slave-06lh krb5kdc[28060](info): set up 2 sockets
Sep 28 14:40:19 dist-test-slave-06lh krb5kdc[28060](info): commencing operation
krb5kdc: starting...
W20260928 14:40:20.711763 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:20 dist-test-slave-06lh krb5kdc[28060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:40:20.838330 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42311
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:42311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:20.951557 28073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:20.951828 28073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:20.951905 28073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:20.956043 28073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:20.956108 28073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:20.956128 28073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:20.956146 28073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:20.956187 28073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:20.961699 28073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:42311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:42311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:20.962860 28073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:20.964037 28073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:20.972895 28079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:20.972901 28078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:20.972898 28081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:20.973405 28073 server_base.cc:1034] running on GCE node
I20260928 14:40:20.974062 28073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:20.975306 28073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:20.976504 28073 hybrid_clock.cc:648] HybridClock initialized: now 1790606420976490 us; error 38 us; skew 500 ppm
Sep 28 14:40:20 dist-test-slave-06lh krb5kdc[28060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606420, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:20.979353 28073 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:40:20.980273 28073 webserver.cc:460] Webserver started at http://127.6.23.254:36299/ using document root <none> and password file <none>
I20260928 14:40:20.980854 28073 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:20.980947 28073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:20.981184 28073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:20.983345 28073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "7a54437b34d44c40835226e0cd82f04a"
format_stamp: "Formatted at 2026-09-28 14:40:20 on dist-test-slave-06lh"
server_key: "da70ac375708d95b9180dc923b65c76a"
server_key_iv: "4d8b132ab854ef79a4bbf0d73c62639c"
server_key_version: "encryptionkey@0"
I20260928 14:40:20.983923 28073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "7a54437b34d44c40835226e0cd82f04a"
format_stamp: "Formatted at 2026-09-28 14:40:20 on dist-test-slave-06lh"
server_key: "da70ac375708d95b9180dc923b65c76a"
server_key_iv: "4d8b132ab854ef79a4bbf0d73c62639c"
server_key_version: "encryptionkey@0"
I20260928 14:40:20.987896 28073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:40:20.990340 28088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:20.991664 28073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:20.991814 28073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "7a54437b34d44c40835226e0cd82f04a"
format_stamp: "Formatted at 2026-09-28 14:40:20 on dist-test-slave-06lh"
server_key: "da70ac375708d95b9180dc923b65c76a"
server_key_iv: "4d8b132ab854ef79a4bbf0d73c62639c"
server_key_version: "encryptionkey@0"
I20260928 14:40:20.991933 28073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:20.996672 28073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:20.999694 28073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:20.999892 28073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:21.006835 28073 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:42311
I20260928 14:40:21.006860 28140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:42311 every 8 connection(s)
I20260928 14:40:21.007818 28073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:21.010576 28141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:21.014829 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28073
I20260928 14:40:21.014968 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:21.015293 6239 external_mini_cluster.cc:1428] Setting key f05a861d7d22f371bbaaf6b8114fed40
I20260928 14:40:21.016552 28141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a: Bootstrap starting.
I20260928 14:40:21.019373 28141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:21.020392 28141 log.cc:824] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:21.021212 28144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:21.016810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53613 (local address 127.6.23.254:42311)
0928 14:40:21.017242 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:21.017271 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:21.017314 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:40:21.018198 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:21.018221 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:21.018596 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:21.018977 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:21.019807 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:21.019812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:21.019828 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:21.019839 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:21.019914 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:21.019940 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:21.020051 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:21.020150 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 14:40:21.020324 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":122,"threads_started":1}
I20260928 14:40:21.022716 28141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a: No bootstrap required, opened a new log
I20260928 14:40:21.024370 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:21.026844 28141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a54437b34d44c40835226e0cd82f04a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42311 } }
I20260928 14:40:21.027184 28141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:21.027295 28141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a54437b34d44c40835226e0cd82f04a, State: Initialized, Role: FOLLOWER
I20260928 14:40:21.027802 28141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a54437b34d44c40835226e0cd82f04a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42311 } }
I20260928 14:40:21.027984 28141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:21.028082 28141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:21.028187 28141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:21.028404 28144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:21.026478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33277 (local address 127.6.23.254:42311)
0928 14:40:21.026617 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:21.026622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:21.026833 (+ 211us) server_negotiation.cc:407] Connection header received
0928 14:40:21.026982 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:21.026984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:21.027015 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:21.027073 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:21.027920 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:21.027923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:21.027924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:21.027935 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:21.027962 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:21.028000 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:21.028089 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:21.028223 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:40:21.028264 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:40:21.029251 28141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a54437b34d44c40835226e0cd82f04a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42311 } }
I20260928 14:40:21.029606 28141 leader_election.cc:302] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a54437b34d44c40835226e0cd82f04a; no voters:
I20260928 14:40:21.029919 28141 leader_election.cc:288] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:21.030057 28151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:21.030350 28151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [term 1 LEADER]: Becoming Leader. State: Replica: 7a54437b34d44c40835226e0cd82f04a, State: Running, Role: LEADER
I20260928 14:40:21.030687 28151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a54437b34d44c40835226e0cd82f04a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42311 } }
I20260928 14:40:21.031363 28141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:21.032584 28152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a54437b34d44c40835226e0cd82f04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a54437b34d44c40835226e0cd82f04a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42311 } } }
I20260928 14:40:21.032584 28153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a54437b34d44c40835226e0cd82f04a. Latest consensus state: current_term: 1 leader_uuid: "7a54437b34d44c40835226e0cd82f04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a54437b34d44c40835226e0cd82f04a" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 42311 } } }
I20260928 14:40:21.032740 28152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:21.032740 28153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:21.033165 28161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:21.036156 28161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:21.042443 28161 catalog_manager.cc:1347] Generated new cluster ID: 0dbae2bc2793497daccf9efd5b8fc8ec
I20260928 14:40:21.042882 28161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:21.051285 28161 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:21.051630 28161 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:21.058967 28161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a54437b34d44c40835226e0cd82f04a: Generated new TSK 0
I20260928 14:40:21.059772 28161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:21.071326 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28073
2026-09-28T14:40:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:21 dist-test-slave-06lh krb5kdc[28172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:21 dist-test-slave-06lh krb5kdc[28172](info): set up 2 sockets
Sep 28 14:40:21 dist-test-slave-06lh krb5kdc[28172](info): commencing operation
krb5kdc: starting...
W20260928 14:40:22.103782 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:22 dist-test-slave-06lh krb5kdc[28172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:40:22.231340 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39381
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:34829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:22.344133 28185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:22.344501 28185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:22.344592 28185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:22.348786 28185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:22.348899 28185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:22.348941 28185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:22.348995 28185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:22.349036 28185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:22.354723 28185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:34829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:39381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:22.355938 28185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:22.357007 28185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:22.365902 28193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:22.366062 28185 server_base.cc:1034] running on GCE node
W20260928 14:40:22.365969 28190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:22.365945 28191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:22.366796 28185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:22.368048 28185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:22.369231 28185 hybrid_clock.cc:648] HybridClock initialized: now 1790606422369220 us; error 41 us; skew 500 ppm
Sep 28 14:40:22 dist-test-slave-06lh krb5kdc[28172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606422, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:22.377707 28185 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:40:22.378732 28185 webserver.cc:460] Webserver started at http://127.6.23.254:44147/ using document root <none> and password file <none>
I20260928 14:40:22.379361 28185 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:22.379431 28185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:22.379608 28185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:22.381655 28185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "7d881966045d4653a3ee06e26488c3d7"
format_stamp: "Formatted at 2026-09-28 14:40:22 on dist-test-slave-06lh"
server_key: "20afbeb774e5be70e4f2a9a38bc12d91"
server_key_iv: "32102558653ccde8b383cef3aa434b61"
server_key_version: "encryptionkey@0"
I20260928 14:40:22.382169 28185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "7d881966045d4653a3ee06e26488c3d7"
format_stamp: "Formatted at 2026-09-28 14:40:22 on dist-test-slave-06lh"
server_key: "20afbeb774e5be70e4f2a9a38bc12d91"
server_key_iv: "32102558653ccde8b383cef3aa434b61"
server_key_version: "encryptionkey@0"
I20260928 14:40:22.385943 28185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:40:22.388381 28200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:22.389640 28185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:22.389783 28185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "7d881966045d4653a3ee06e26488c3d7"
format_stamp: "Formatted at 2026-09-28 14:40:22 on dist-test-slave-06lh"
server_key: "20afbeb774e5be70e4f2a9a38bc12d91"
server_key_iv: "32102558653ccde8b383cef3aa434b61"
server_key_version: "encryptionkey@0"
I20260928 14:40:22.389892 28185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:22.411186 28185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:22.414361 28185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:22.414580 28185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:22.422169 28185 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:39381
I20260928 14:40:22.422197 28252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:39381 every 8 connection(s)
I20260928 14:40:22.423163 28185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:22.426015 28253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:22.428082 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28185
I20260928 14:40:22.428241 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:22.428555 6239 external_mini_cluster.cc:1428] Setting key 0a85949d5ecf945aced88389a1eb07bb
I20260928 14:40:22.432814 28253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7: Bootstrap starting.
I20260928 14:40:22.435134 28253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:22.435989 28253 log.cc:824] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:40:22 dist-test-slave-06lh krb5kdc[28172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:22.438019 28253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7: No bootstrap required, opened a new log
I20260928 14:40:22.442525 28253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d881966045d4653a3ee06e26488c3d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39381 } }
I20260928 14:40:22.442813 28253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:22.442875 28253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d881966045d4653a3ee06e26488c3d7, State: Initialized, Role: FOLLOWER
I20260928 14:40:22.443378 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:22.430046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36707 (local address 127.6.23.254:39381)
0928 14:40:22.430513 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:22.430529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:22.430567 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:40:22.431286 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:22.431312 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:22.431659 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:22.432000 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:22.432855 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:22.433876 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:22.434766 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:22.435320 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:22.435485 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:22.438041 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:22.438045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:22.438059 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:22.438087 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:22.439952 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:22.440521 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:22.440533 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:22.440540 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:22.440714 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:22.441011 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:22.441015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:22.441016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:22.441448 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:22.441685 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:22.441998 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 14:40:22.442331 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":147,"threads_started":1}
I20260928 14:40:22.443377 28253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d881966045d4653a3ee06e26488c3d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39381 } }
I20260928 14:40:22.443662 28253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:22.443742 28253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:22.443845 28253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:22.445034 28253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d881966045d4653a3ee06e26488c3d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39381 } }
I20260928 14:40:22.445443 28253 leader_election.cc:302] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d881966045d4653a3ee06e26488c3d7; no voters:
I20260928 14:40:22.445843 28253 leader_election.cc:288] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:22.445955 28258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:22.446194 28258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [term 1 LEADER]: Becoming Leader. State: Replica: 7d881966045d4653a3ee06e26488c3d7, State: Running, Role: LEADER
I20260928 14:40:22.446578 28258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d881966045d4653a3ee06e26488c3d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39381 } }
I20260928 14:40:22.446647 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:22.447351 28253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:22.448323 28260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d881966045d4653a3ee06e26488c3d7. Latest consensus state: current_term: 1 leader_uuid: "7d881966045d4653a3ee06e26488c3d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d881966045d4653a3ee06e26488c3d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39381 } } }
I20260928 14:40:22.448462 28260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:22.448812 28259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d881966045d4653a3ee06e26488c3d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d881966045d4653a3ee06e26488c3d7" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 39381 } } }
I20260928 14:40:22.448905 28259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:22.449021 28271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:22.452857 28271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:22.459556 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:22.450440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48857 (local address 127.6.23.254:39381)
0928 14:40:22.450559 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:22.450564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:22.450775 (+ 211us) server_negotiation.cc:407] Connection header received
0928 14:40:22.450937 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:22.450941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:22.450994 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:22.451113 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:22.452023 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:22.452842 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:22.454035 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:22.454303 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:22.454420 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:22.455906 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:22.455910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:22.455913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:22.455938 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:22.457427 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:22.457863 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:22.457866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:22.457869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:22.457956 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:22.458246 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:22.458250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:22.458253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:22.458472 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:22.458591 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:22.459279 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 14:40:22.459399 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:40:22.460060 28271 catalog_manager.cc:1347] Generated new cluster ID: f1f1f95e90dc4091b7883641fb639aae
I20260928 14:40:22.460140 28271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:22.468863 28271 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:22.469787 28271 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:22.482191 28271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d881966045d4653a3ee06e26488c3d7: Generated new TSK 0
I20260928 14:40:22.483105 28271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:22.492867 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28185
2026-09-28T14:40:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:22 dist-test-slave-06lh krb5kdc[28284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:22 dist-test-slave-06lh krb5kdc[28284](info): set up 2 sockets
Sep 28 14:40:22 dist-test-slave-06lh krb5kdc[28284](info): commencing operation
krb5kdc: starting...
W20260928 14:40:23.527858 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:23 dist-test-slave-06lh krb5kdc[28284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:40:23.654659 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35945
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:35945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:23.767838 28297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:23.768231 28297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:23.768323 28297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:23.772413 28297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:23.772524 28297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:23.772563 28297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:23.772601 28297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:23.772635 28297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:23.778178 28297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:35945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:23.779453 28297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:23.780561 28297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:23.789549 28305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:23.789707 28303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:23.789599 28297 server_base.cc:1034] running on GCE node
W20260928 14:40:23.789638 28302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:23.790432 28297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:23.791698 28297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:23.792870 28297 hybrid_clock.cc:648] HybridClock initialized: now 1790606423792845 us; error 41 us; skew 500 ppm
Sep 28 14:40:23 dist-test-slave-06lh krb5kdc[28284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:23.795811 28297 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:40:23.796675 28297 webserver.cc:460] Webserver started at http://127.6.23.254:42209/ using document root <none> and password file <none>
I20260928 14:40:23.797214 28297 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:23.797273 28297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:23.797437 28297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:23.799539 28297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "4bb3850c1938425d8832ca1e77e906d2"
format_stamp: "Formatted at 2026-09-28 14:40:23 on dist-test-slave-06lh"
server_key: "a20ed99d25b61d934ec2a79766eb574d"
server_key_iv: "1691c5a3fb262f056059bfa96f977be3"
server_key_version: "encryptionkey@0"
I20260928 14:40:23.800056 28297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "4bb3850c1938425d8832ca1e77e906d2"
format_stamp: "Formatted at 2026-09-28 14:40:23 on dist-test-slave-06lh"
server_key: "a20ed99d25b61d934ec2a79766eb574d"
server_key_iv: "1691c5a3fb262f056059bfa96f977be3"
server_key_version: "encryptionkey@0"
I20260928 14:40:23.803879 28297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:40:23.806442 28312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:23.807865 28297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:40:23.807974 28297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "4bb3850c1938425d8832ca1e77e906d2"
format_stamp: "Formatted at 2026-09-28 14:40:23 on dist-test-slave-06lh"
server_key: "a20ed99d25b61d934ec2a79766eb574d"
server_key_iv: "1691c5a3fb262f056059bfa96f977be3"
server_key_version: "encryptionkey@0"
I20260928 14:40:23.808065 28297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:23.839617 28297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:23.842994 28297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:23.843165 28297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:23.850677 28297 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:35945
I20260928 14:40:23.850687 28364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:35945 every 8 connection(s)
I20260928 14:40:23.851747 28297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:23.854750 28365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:23.860656 28365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2: Bootstrap starting.
I20260928 14:40:23.861824 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28297
I20260928 14:40:23.862002 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:23.862314 6239 external_mini_cluster.cc:1428] Setting key 8824f3b70f9c37b964e88dbd4cc17d67
I20260928 14:40:23.863873 28365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:23.864637 28365 log.cc:824] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:23.866792 28365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2: No bootstrap required, opened a new log
I20260928 14:40:23.871428 28365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb3850c1938425d8832ca1e77e906d2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35945 } }
I20260928 14:40:23.871706 28365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:23.871790 28365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb3850c1938425d8832ca1e77e906d2, State: Initialized, Role: FOLLOWER
I20260928 14:40:23.871886 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:23.863556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46201 (local address 127.6.23.254:35945)
0928 14:40:23.863990 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:23.864013 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:23.864066 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:40:23.864738 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:23.864785 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:23.865255 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:23.865603 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:23.865937 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:23.866929 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:23.868222 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:23.868733 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:23.868862 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:23.869763 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:23.869768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:23.869783 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:23.870239 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:23.870298 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:23.870323 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:23.870400 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:23.870603 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:40:23.870873 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":140,"threads_started":1}
I20260928 14:40:23.872354 28365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb3850c1938425d8832ca1e77e906d2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35945 } }
I20260928 14:40:23.872524 28365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:23.872578 28365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:23.872674 28365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:23.873728 28365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb3850c1938425d8832ca1e77e906d2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35945 } }
I20260928 14:40:23.874135 28365 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bb3850c1938425d8832ca1e77e906d2; no voters:
I20260928 14:40:23.874518 28365 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:23.874635 28370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:23.874862 28370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb3850c1938425d8832ca1e77e906d2, State: Running, Role: LEADER
I20260928 14:40:23.874953 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:23.875253 28370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb3850c1938425d8832ca1e77e906d2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35945 } }
I20260928 14:40:23.875955 28365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:23.876741 28371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bb3850c1938425d8832ca1e77e906d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb3850c1938425d8832ca1e77e906d2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35945 } } }
I20260928 14:40:23.876865 28371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:23.877195 28372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb3850c1938425d8832ca1e77e906d2. Latest consensus state: current_term: 1 leader_uuid: "4bb3850c1938425d8832ca1e77e906d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb3850c1938425d8832ca1e77e906d2" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35945 } } }
I20260928 14:40:23.877285 28372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:23.877417 28383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:23.881026 28383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:23.882901 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:23.877883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48677 (local address 127.6.23.254:35945)
0928 14:40:23.878000 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:23.878005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:23.878235 (+ 230us) server_negotiation.cc:407] Connection header received
0928 14:40:23.878443 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:23.878446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:23.878498 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:23.878548 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:23.879017 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:23.879839 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:23.881027 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:23.881363 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:23.881471 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:23.882474 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:23.882478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:23.882481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:23.882493 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:23.882529 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:23.882540 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:23.882622 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:23.882712 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 14:40:23.882765 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:40:23.887319 28383 catalog_manager.cc:1347] Generated new cluster ID: 0c8755803fa944eb8696f6d1c827bdfd
I20260928 14:40:23.887400 28383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:23.909509 28383 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:23.910560 28383 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:23.916411 28383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bb3850c1938425d8832ca1e77e906d2: Generated new TSK 0
I20260928 14:40:23.917168 28383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:23.927661 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28297
2026-09-28T14:40:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:23 dist-test-slave-06lh krb5kdc[28396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:23 dist-test-slave-06lh krb5kdc[28396](info): set up 2 sockets
Sep 28 14:40:23 dist-test-slave-06lh krb5kdc[28396](info): commencing operation
krb5kdc: starting...
W20260928 14:40:24.971782 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:25 dist-test-slave-06lh krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:40:25.098431 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45203
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:25.212682 28409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:25.212981 28409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:25.213025 28409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:25.217216 28409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:25.217291 28409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:25.217311 28409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:25.217329 28409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:25.217343 28409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:25.222862 28409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:25.224028 28409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:25.225102 28409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:40:25.234030 28409 server_base.cc:1034] running on GCE node
W20260928 14:40:25.233994 28417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:25.234117 28414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:25.234053 28415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:25.234969 28409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:25.236253 28409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:25.237426 28409 hybrid_clock.cc:648] HybridClock initialized: now 1790606425237395 us; error 49 us; skew 500 ppm
Sep 28 14:40:25 dist-test-slave-06lh krb5kdc[28396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606425, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:25.240330 28409 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:40:25.241329 28409 webserver.cc:460] Webserver started at http://127.6.23.254:45597/ using document root <none> and password file <none>
I20260928 14:40:25.241918 28409 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:25.242012 28409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:25.242257 28409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:25.244447 28409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "067063b884534a98bb87365cf8ccd142"
format_stamp: "Formatted at 2026-09-28 14:40:25 on dist-test-slave-06lh"
server_key: "63af0a039d09c191b1055725881abd19"
server_key_iv: "869ace2306f6b10b62b4400c2ef17026"
server_key_version: "encryptionkey@0"
I20260928 14:40:25.245046 28409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "067063b884534a98bb87365cf8ccd142"
format_stamp: "Formatted at 2026-09-28 14:40:25 on dist-test-slave-06lh"
server_key: "63af0a039d09c191b1055725881abd19"
server_key_iv: "869ace2306f6b10b62b4400c2ef17026"
server_key_version: "encryptionkey@0"
I20260928 14:40:25.249083 28409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:40:25.251767 28424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:25.253017 28409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:25.253201 28409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "067063b884534a98bb87365cf8ccd142"
format_stamp: "Formatted at 2026-09-28 14:40:25 on dist-test-slave-06lh"
server_key: "63af0a039d09c191b1055725881abd19"
server_key_iv: "869ace2306f6b10b62b4400c2ef17026"
server_key_version: "encryptionkey@0"
I20260928 14:40:25.253326 28409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:25.258003 28409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:25.261016 28409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:25.261228 28409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:25.268517 28409 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:45203
I20260928 14:40:25.268543 28476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:45203 every 8 connection(s)
I20260928 14:40:25.269532 28409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:25.272351 28477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:25.274842 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28409
I20260928 14:40:25.274987 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:25.275315 6239 external_mini_cluster.cc:1428] Setting key 49852029b723ebbb9b2f7d0fa2309733
I20260928 14:40:25.278492 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142: Bootstrap starting.
I20260928 14:40:25.280553 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:25.276772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39665 (local address 127.6.23.254:45203)
0928 14:40:25.277225 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:25.277240 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:25.277274 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:40:25.277899 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:25.277917 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:25.278205 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:25.278554 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:25.279372 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:25.279377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:25.279389 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:25.279400 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:25.279453 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:25.279473 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:25.279633 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:25.279703 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 14:40:25.279865 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":129,"threads_started":1}
I20260928 14:40:25.281101 28477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:25.281901 28477 log.cc:824] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:25.283509 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:25.284068 28477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142: No bootstrap required, opened a new log
I20260928 14:40:25.287511 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:25.285660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59847 (local address 127.6.23.254:45203)
0928 14:40:25.285776 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:25.285779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:25.285996 (+ 217us) server_negotiation.cc:407] Connection header received
0928 14:40:25.286176 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:25.286179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:25.286213 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:25.286308 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:25.287021 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:25.287025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:25.287027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:40:25.287038 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:25.287064 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:40:25.287092 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:25.287188 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:25.287340 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 14:40:25.287377 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:40:25.288543 28477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067063b884534a98bb87365cf8ccd142" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45203 } }
I20260928 14:40:25.288827 28477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:40:25.288895 28477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 067063b884534a98bb87365cf8ccd142, State: Initialized, Role: FOLLOWER
I20260928 14:40:25.289317 28477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067063b884534a98bb87365cf8ccd142" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45203 } }
I20260928 14:40:25.289462 28477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:40:25.289530 28477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:40:25.289630 28477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:25.290577 28477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067063b884534a98bb87365cf8ccd142" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45203 } }
I20260928 14:40:25.290930 28477 leader_election.cc:302] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 067063b884534a98bb87365cf8ccd142; no voters:
I20260928 14:40:25.291301 28477 leader_election.cc:288] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:40:25.291415 28487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:25.291659 28487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [term 1 LEADER]: Becoming Leader. State: Replica: 067063b884534a98bb87365cf8ccd142, State: Running, Role: LEADER
I20260928 14:40:25.292327 28487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067063b884534a98bb87365cf8ccd142" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45203 } }
I20260928 14:40:25.292695 28477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:25.294027 28488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "067063b884534a98bb87365cf8ccd142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067063b884534a98bb87365cf8ccd142" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45203 } } }
I20260928 14:40:25.294155 28488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:25.294484 28489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 067063b884534a98bb87365cf8ccd142. Latest consensus state: current_term: 1 leader_uuid: "067063b884534a98bb87365cf8ccd142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "067063b884534a98bb87365cf8ccd142" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45203 } } }
I20260928 14:40:25.294585 28489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142 [sys.catalog]: This master's current role is: LEADER
W20260928 14:40:25.297487 28502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:25.297554 28502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:25.297638 28497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:25.298471 28497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:25.303985 28497 catalog_manager.cc:1347] Generated new cluster ID: 0ee828fce0d64ea28a52a1d6d627fe46
I20260928 14:40:25.304188 28497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:25.314875 28497 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:25.315295 28497 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:25.326402 28497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 067063b884534a98bb87365cf8ccd142: Generated new TSK 0
I20260928 14:40:25.327554 28497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:40:25.342018 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28409
2026-09-28T14:40:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1414 ms)
[----------] 18 tests from AuthTokenIssuingTest (25598 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:25 dist-test-slave-06lh krb5kdc[28508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:25 dist-test-slave-06lh krb5kdc[28508](info): set up 2 sockets
Sep 28 14:40:25 dist-test-slave-06lh krb5kdc[28508](info): commencing operation
krb5kdc: starting...
W20260928 14:40:26.383968 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:26 dist-test-slave-06lh krb5kdc[28508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:40:26.514461 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45981
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45981,127.6.23.253:43153,127.6.23.252:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:26.633591 28521 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:40:26.634006 28521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:26.634083 28521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:26.634163 28521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:26.638386 28521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:26.638504 28521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:26.638541 28521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:26.638582 28521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:26.638619 28521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:26.644191 28521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45981,127.6.23.253:43153,127.6.23.252:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:45981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:26.645395 28521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:26.646497 28521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:26.655540 28526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:26.655565 28527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:26.655570 28529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:26.655926 28521 server_base.cc:1034] running on GCE node
I20260928 14:40:26.656553 28521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:26.657727 28521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:26.658934 28521 hybrid_clock.cc:648] HybridClock initialized: now 1790606426658912 us; error 35 us; skew 500 ppm
Sep 28 14:40:26 dist-test-slave-06lh krb5kdc[28508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:26.662029 28521 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:40:26.663060 28521 webserver.cc:460] Webserver started at http://127.6.23.254:38379/ using document root <none> and password file <none>
I20260928 14:40:26.663724 28521 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:26.663818 28521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:26.664054 28521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:26.666215 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "f2e6726b8995415694bf29d99062287e"
format_stamp: "Formatted at 2026-09-28 14:40:26 on dist-test-slave-06lh"
server_key: "95bca34eb85e2a44b273af40b7cf013c"
server_key_iv: "ae9ea523027bf8e8072367963633a368"
server_key_version: "encryptionkey@0"
I20260928 14:40:26.666829 28521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "f2e6726b8995415694bf29d99062287e"
format_stamp: "Formatted at 2026-09-28 14:40:26 on dist-test-slave-06lh"
server_key: "95bca34eb85e2a44b273af40b7cf013c"
server_key_iv: "ae9ea523027bf8e8072367963633a368"
server_key_version: "encryptionkey@0"
I20260928 14:40:26.670966 28521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 14:40:26.673743 28536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:26.675112 28521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:40:26.675338 28521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "f2e6726b8995415694bf29d99062287e"
format_stamp: "Formatted at 2026-09-28 14:40:26 on dist-test-slave-06lh"
server_key: "95bca34eb85e2a44b273af40b7cf013c"
server_key_iv: "ae9ea523027bf8e8072367963633a368"
server_key_version: "encryptionkey@0"
I20260928 14:40:26.675477 28521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:26.697088 28521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:26.700497 28521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:26.700727 28521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:26.708215 28521 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:45981
I20260928 14:40:26.708621 28588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:45981 every 8 connection(s)
I20260928 14:40:26.709282 28521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:26.711892 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28521
I20260928 14:40:26.712044 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:26.712355 6239 external_mini_cluster.cc:1428] Setting key bf9689649274006e9859856a9de52b16
I20260928 14:40:26.712589 28589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:26.717022 28589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.23.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.253@KRBTEST.COM" created.
Sep 28 14:40:26 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:26.734331 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.720817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.6.23.254:45981)
0928 14:40:26.721113 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:26.721127 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:26.721162 (+ 35us) server_negotiation.cc:407] Connection header received
0928 14:40:26.721637 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:26.721642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:26.721834 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:26.721995 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:26.722789 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.724174 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.725269 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.725943 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.726133 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:26.728912 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:26.728921 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:26.728940 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:26.728974 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:26.731043 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:26.731611 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:26.731619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:26.731622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:26.731793 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:26.732121 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:26.732125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:26.732126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:26.732701 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:26.732834 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:26.733185 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 14:40:26.733435 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":189,"spinlock_wait_cycles":4480,"thread_start_us":115,"threads_started":1}
I20260928 14:40:26.734338 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.719732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45981 (local address 127.0.0.1:36974)
0928 14:40:26.720343 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.720398 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:26.721328 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:26.722027 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:26.722032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:26.722379 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:26.722632 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:26.722645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.724341 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.724344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:26.725138 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:26.725148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.726118 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.726121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:26.726249 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:26.726888 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:26.726905 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:26.728734 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:26.731187 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:26.731196 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:26.731206 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:26.731490 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:26.731902 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:26.731904 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:26.731905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:26.732014 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:26.732816 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:26.732821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:26.732945 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 14:40:26.733082 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 14:40:26.733487 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":95,"spinlock_wait_cycles":7808,"thread_start_us":163,"threads_started":1}
I20260928 14:40:26.737160 28589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:26.737975 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.737703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48662)
0928 14:40:26.737830 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.737911 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 14:40:26.738207 28539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
I20260928 14:40:26.739042 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.738828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48672)
0928 14:40:26.738936 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.738987 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:40:26.739271 28589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:43153: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab.
Entry for principal kudu/127.6.23.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab.
I20260928 14:40:26.742486 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.253:43153
--webserver_interface=127.6.23.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab
--principal=kudu/127.6.23.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45981,127.6.23.253:43153,127.6.23.252:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:40:26.788560 28589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } attempt: 1
I20260928 14:40:26.789492 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.789181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48674)
0928 14:40:26.789343 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.789410 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 14:40:26.790346 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.790116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48688)
0928 14:40:26.790238 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.790287 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 14:40:26.790556 28589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:43153: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
I20260928 14:40:26.858767 28589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } attempt: 2
I20260928 14:40:26.859714 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.859385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48696)
0928 14:40:26.859547 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.859631 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:40:26.860507 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.860312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48708)
0928 14:40:26.860409 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.860453 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 14:40:26.860756 28589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:43153: Network error: Client connection negotiation failed: client connection to 127.6.23.253:43153: connect: Connection refused (error 111)
W20260928 14:40:26.861024 28596 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:40:26.861351 28596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:26.861388 28596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:26.861497 28596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:26.865759 28596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:26.865825 28596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:26.865845 28596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:26.865886 28596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:26.865921 28596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:26.871712 28596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45981,127.6.23.253:43153,127.6.23.252:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab
--principal=kudu/127.6.23.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.253:43153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.23.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:26.872903 28596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:26.874006 28596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:26.883157 28602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:26.883148 28601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:26.883141 28604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:26.883571 28596 server_base.cc:1034] running on GCE node
I20260928 14:40:26.884150 28596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:26.885257 28596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:26.886422 28596 hybrid_clock.cc:648] HybridClock initialized: now 1790606426886348 us; error 97 us; skew 500 ppm
Sep 28 14:40:26 dist-test-slave-06lh krb5kdc[28508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:26.889364 28596 init.cc:377] Logged in from keytab as kudu/127.6.23.253@KRBTEST.COM (short username kudu)
I20260928 14:40:26.890256 28596 webserver.cc:460] Webserver started at http://127.6.23.253:41063/ using document root <none> and password file <none>
I20260928 14:40:26.890830 28596 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:26.890890 28596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:26.891063 28596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:26.893213 28596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/data/instance:
uuid: "1015335f40f94f359ede0ce18531aeb6"
format_stamp: "Formatted at 2026-09-28 14:40:26 on dist-test-slave-06lh"
server_key: "eefdee385bff1e0689ef03af1ca9338f"
server_key_iv: "33280e236efe74c9b7b7e249c73d2ee8"
server_key_version: "encryptionkey@0"
I20260928 14:40:26.893751 28596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/wal/instance:
uuid: "1015335f40f94f359ede0ce18531aeb6"
format_stamp: "Formatted at 2026-09-28 14:40:26 on dist-test-slave-06lh"
server_key: "eefdee385bff1e0689ef03af1ca9338f"
server_key_iv: "33280e236efe74c9b7b7e249c73d2ee8"
server_key_version: "encryptionkey@0"
I20260928 14:40:26.897617 28596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:40:26.900175 28611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:26.901504 28596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:26.901623 28596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/wal
uuid: "1015335f40f94f359ede0ce18531aeb6"
format_stamp: "Formatted at 2026-09-28 14:40:26 on dist-test-slave-06lh"
server_key: "eefdee385bff1e0689ef03af1ca9338f"
server_key_iv: "33280e236efe74c9b7b7e249c73d2ee8"
server_key_version: "encryptionkey@0"
I20260928 14:40:26.901718 28596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:26.918617 28596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:26.921913 28596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:26.922086 28596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:26.929438 28596 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.253:43153
I20260928 14:40:26.929761 28663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.253:43153 every 8 connection(s)
I20260928 14:40:26.930541 28596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
I20260928 14:40:26.933549 28664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:26.937470 28664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:26.939549 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28596
I20260928 14:40:26.939679 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-1/wal/instance
I20260928 14:40:26.940009 6239 external_mini_cluster.cc:1428] Setting key c4d7c41271d5342ca3c52985368319a5
WARNING: no policy specified for kudu/127.6.23.252@KRBTEST.COM; defaulting to no policy
Sep 28 14:40:26 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.252@KRBTEST.COM" created.
I20260928 14:40:26.951932 28589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } attempt: 3
I20260928 14:40:26.953529 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.939560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36990 (local address 127.6.23.254:45981)
0928 14:40:26.939714 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:26.939722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:26.940867 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 14:40:26.942098 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:26.942102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:26.942154 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:26.942234 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:26.944037 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.944956 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.946237 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.946664 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.946793 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:26.949286 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:26.949289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:26.949292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:26.949327 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:26.950850 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:26.951448 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:26.951450 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:26.951452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:26.951532 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:26.952006 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:26.952009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:26.952012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:26.952280 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:26.952412 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:26.953191 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 14:40:26.953327 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:40:26.954737 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.940129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45981 (local address 127.0.0.1:36990)
0928 14:40:26.940706 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.940753 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:26.941831 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:26.942471 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:26.942481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:26.943030 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:26.943839 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:26.943865 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.945115 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.945118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:26.946085 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:26.946095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.946834 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.946837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:26.946989 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:26.947677 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:26.947694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:26.949109 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:26.950971 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:26.950976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:26.950990 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:26.951332 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:26.951670 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:26.951676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:26.951681 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:26.951888 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:26.952479 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:26.952490 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:26.952874 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 14:40:26.953099 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 14:40:26.953517 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":186,"threads_started":1}
I20260928 14:40:26.956171 28664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:40:26 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
Sep 28 14:40:26 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:40:26.964085 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.952457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48712)
0928 14:40:26.952649 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.952668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:26.952749 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:26.953465 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:26.953468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:26.953645 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:26.954209 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:26.954218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.955445 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.955451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:26.956417 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:26.956429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.957372 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.957380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:26.957540 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:26.958120 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:26.958141 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:26.959799 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:26.962258 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:26.962264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:26.962269 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:26.962499 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:26.962799 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:26.962801 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:26.962803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:26.962899 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:26.963503 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:26.963505 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:26.963624 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:40:26.963736 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:40:26.963900 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 14:40:26.964536 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.952565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48712 (local address 127.6.23.253:43153)
0928 14:40:26.952983 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:26.952992 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:26.953010 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:40:26.953046 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:26.953050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:26.953317 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:26.954065 (+ 748us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:26.954339 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.955289 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.956574 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.957204 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.957806 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:26.959958 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:26.959971 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:26.959989 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:26.960022 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:26.962109 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:26.962614 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:26.962620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:26.962621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:26.962709 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:26.963058 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:26.963063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:26.963064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:26.963408 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:26.963548 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:26.963835 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 14:40:26.963987 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":122,"threads_started":1}
I20260928 14:40:26.966895 28589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:26.967852 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.967545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59044)
0928 14:40:26.967679 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.967763 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 14:40:26.968362 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.956733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48724)
0928 14:40:26.956932 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.956947 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:26.957025 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:26.957284 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:26.957287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:26.957457 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:26.957533 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:26.957537 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.958478 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.958483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:26.959338 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:26.959351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.960030 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.960033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:26.960094 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:26.960560 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:26.960578 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:26.964445 (+ 3867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:26.966555 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:26.966559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:26.966563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:26.966844 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:26.967154 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:26.967158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:26.967160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:26.967327 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:26.967777 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:26.967780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:26.967876 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 14:40:26.968032 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:40:26.968169 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260928 14:40:26.968406 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.956771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48724 (local address 127.6.23.253:43153)
0928 14:40:26.957083 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:26.957095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:26.957108 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:40:26.957136 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:26.957139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:26.957172 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:26.957271 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:26.957639 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.958349 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.959476 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:26.959835 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:26.959929 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:26.964599 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:26.964601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:26.964603 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:26.964647 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:26.966437 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:26.966964 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:26.966968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:26.966970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:26.967059 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:26.967431 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:26.967434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:26.967435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:26.967671 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:26.967771 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:26.968122 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 14:40:26.968251 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260928 14:40:26.968724 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.968389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59054)
0928 14:40:26.968581 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.968631 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260928 14:40:26.968838 28589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:43703: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
I20260928 14:40:26.969133 28664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:26.969754 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.969531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59058)
0928 14:40:26.969632 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.969703 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab.
Entry for principal kudu/127.6.23.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab.
W20260928 14:40:26.970005 28614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
I20260928 14:40:26.970878 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:26.970668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59062)
0928 14:40:26.970782 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:26.970832 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 14:40:26.970983 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.252:43703
--webserver_interface=127.6.23.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab
--principal=kudu/127.6.23.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:41345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:45981,127.6.23.253:43153,127.6.23.252:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:26.971033 28664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:43703: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
I20260928 14:40:27.000021 28589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } attempt: 1
I20260928 14:40:27.000932 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.000628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59076)
0928 14:40:27.000781 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.000854 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 14:40:27.001770 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.001519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59078)
0928 14:40:27.001662 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.001712 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 14:40:27.001963 28589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:43703: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
I20260928 14:40:27.020282 28664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } attempt: 1
I20260928 14:40:27.021226 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.020858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59090)
0928 14:40:27.021035 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.021114 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 14:40:27.022038 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.021797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59104)
0928 14:40:27.021913 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.021970 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:40:27.022238 28664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:43703: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
I20260928 14:40:27.077155 28589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } attempt: 2
I20260928 14:40:27.078011 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.077707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59108)
0928 14:40:27.077857 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.077934 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:40:27.078785 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.078519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59122)
0928 14:40:27.078640 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.078715 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:40:27.078979 28589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:43703: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
W20260928 14:40:27.088455 28672 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:40:27.088827 28672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:27.088892 28672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:27.088976 28672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:40:27.090453 28664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } attempt: 2
I20260928 14:40:27.091336 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.090966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59134)
0928 14:40:27.091140 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.091214 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:40:27.092181 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.091944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59142)
0928 14:40:27.092060 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.092114 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:40:27.092340 28664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:43703: Network error: Client connection negotiation failed: client connection to 127.6.23.252:43703: connect: Connection refused (error 111)
W20260928 14:40:27.093653 28672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:27.093706 28672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:27.093726 28672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:27.093743 28672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:27.093760 28672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:27.099337 28672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:41345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:45981,127.6.23.253:43153,127.6.23.252:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab
--principal=kudu/127.6.23.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.252:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.23.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:27.100427 28672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:27.101431 28672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:27.110152 28678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:27.110175 28677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:27.110221 28680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:27.110384 28672 server_base.cc:1034] running on GCE node
I20260928 14:40:27.111194 28672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:27.112437 28672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:27.113611 28672 hybrid_clock.cc:648] HybridClock initialized: now 1790606427113593 us; error 34 us; skew 500 ppm
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606427, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:27.116650 28672 init.cc:377] Logged in from keytab as kudu/127.6.23.252@KRBTEST.COM (short username kudu)
I20260928 14:40:27.117612 28672 webserver.cc:460] Webserver started at http://127.6.23.252:39105/ using document root <none> and password file <none>
I20260928 14:40:27.118166 28672 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:27.118227 28672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:27.118400 28672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:27.120599 28672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/data/instance:
uuid: "2983e92ed2164ca6b5f3a63a400b6a26"
format_stamp: "Formatted at 2026-09-28 14:40:27 on dist-test-slave-06lh"
server_key: "ed35aec37b2ffd577e5a3aab24c6ef2c"
server_key_iv: "80c3658f52ef5d31f7671ee27710013d"
server_key_version: "encryptionkey@0"
I20260928 14:40:27.121318 28672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/wal/instance:
uuid: "2983e92ed2164ca6b5f3a63a400b6a26"
format_stamp: "Formatted at 2026-09-28 14:40:27 on dist-test-slave-06lh"
server_key: "ed35aec37b2ffd577e5a3aab24c6ef2c"
server_key_iv: "80c3658f52ef5d31f7671ee27710013d"
server_key_version: "encryptionkey@0"
I20260928 14:40:27.125295 28672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:40:27.127983 28687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:27.129297 28672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:40:27.129452 28672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/wal
uuid: "2983e92ed2164ca6b5f3a63a400b6a26"
format_stamp: "Formatted at 2026-09-28 14:40:27 on dist-test-slave-06lh"
server_key: "ed35aec37b2ffd577e5a3aab24c6ef2c"
server_key_iv: "80c3658f52ef5d31f7671ee27710013d"
server_key_version: "encryptionkey@0"
I20260928 14:40:27.129576 28672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:27.142031 28672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:27.145488 28672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:27.145721 28672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:27.153399 28672 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.252:43703
I20260928 14:40:27.153683 28739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.252:43703 every 8 connection(s)
I20260928 14:40:27.154479 28672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
I20260928 14:40:27.157437 28740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:27.157852 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28672
I20260928 14:40:27.158002 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606142807864-6239-0/minicluster-data/master-2/wal/instance
I20260928 14:40:27.158279 6239 external_mini_cluster.cc:1428] Setting key c71f84e95105d77d547010810eecc506
I20260928 14:40:27.162678 28740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606427, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:27.174749 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.159766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.6.23.252:43703)
0928 14:40:27.160191 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.160212 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.160290 (+ 78us) server_negotiation.cc:407] Connection header received
0928 14:40:27.161042 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.161066 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.161556 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.161935 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.162709 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.164193 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.165491 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.166153 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.166318 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.169054 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.169060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.169072 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.169103 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.171276 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.171846 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.171853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.171860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.171977 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.172271 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.172277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.172282 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.172708 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.172932 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.173228 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.173566 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260928 14:40:27.174983 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.164814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:45981 (local address 127.0.0.1:37002)
0928 14:40:27.165239 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.165289 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:27.165440 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:27.165869 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:27.165875 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:27.166374 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:27.166533 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.166539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.167681 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.167683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.168329 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.168338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.168932 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.168936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.169042 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.169608 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:27.169628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:27.171465 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:27.173322 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.173329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.173356 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.173648 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.173930 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.173932 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.173933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.174001 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.174413 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:27.174418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:27.174514 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 14:40:27.174626 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 14:40:27.174765 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":149,"threads_started":1}
I20260928 14:40:27.175001 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.164886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.6.23.254:45981)
0928 14:40:27.165051 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.165056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.165372 (+ 316us) server_negotiation.cc:407] Connection header received
0928 14:40:27.165625 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.165628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.165692 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.165795 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.166696 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.167549 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.168475 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.168762 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.168873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.171605 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.171608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.171610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.171637 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.173179 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.173746 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.173749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.173751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.173842 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.174083 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.174086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.174088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.174310 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.174411 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.174721 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.174860 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:40:27.176146 28740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:27.178128 28589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } attempt: 3
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606427, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:27.183481 28664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } attempt: 3
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
I20260928 14:40:27.188143 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.176700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:43153 (local address 127.0.0.1:48726)
0928 14:40:27.177024 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.177040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:27.177127 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:27.177395 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:27.177397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:27.177538 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:27.177675 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.177682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.178791 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.178794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.179692 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.179701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.180421 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.180424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.180522 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.181614 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:27.181628 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:27.183102 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:27.185237 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.185242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.185245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.185762 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.186888 (+ 1126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.186893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.186896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.186993 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.187574 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:27.187580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:27.187675 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 14:40:27.187797 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:40:27.187935 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260928 14:40:27.189056 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.176802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48726 (local address 127.6.23.253:43153)
0928 14:40:27.176925 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.176928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.177094 (+ 166us) server_negotiation.cc:407] Connection header received
0928 14:40:27.177212 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.177215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.177283 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.177368 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.177797 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.178669 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.179807 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.180170 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.180268 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.183313 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.183317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.183322 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.183354 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.185132 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.185894 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.185899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.185903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.186500 (+ 597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.187152 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.187156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.187158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.187432 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.187531 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.188714 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.188883 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:40:27.189275 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.178517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59162)
0928 14:40:27.178668 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.178680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:27.178769 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:27.178988 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:27.178991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:27.179156 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:27.179302 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.179308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.180233 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.180237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.180921 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.180933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.181515 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.181518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.181580 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.182018 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:27.182029 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:27.185542 (+ 3513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:27.187644 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.187647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.187649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.187913 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.188195 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.188198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.188200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.188303 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.188779 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:27.188782 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:27.188877 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 14:40:27.188974 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:40:27.189108 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 14:40:27.189548 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.177530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.6.23.254:45981)
0928 14:40:27.177671 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.177675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.177689 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:40:27.177725 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.177729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.177788 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.177861 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.178668 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.179450 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.180409 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.180749 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.180866 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.185122 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.185126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.185128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.185157 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.187134 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.187655 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.187659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.187660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.187779 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.188080 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.188086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.188091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.188333 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.188595 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.189271 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.189414 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:40:27.189801 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.178556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.6.23.252:43703)
0928 14:40:27.178650 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.178654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.178721 (+ 67us) server_negotiation.cc:407] Connection header received
0928 14:40:27.178851 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.178854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.178894 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.178974 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.179416 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.180111 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.181024 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.181399 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.181501 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.185697 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.185701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.185707 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.185736 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.187535 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.188019 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.188022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.188024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.188097 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.188414 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.188417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.188421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.188640 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.188732 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.189526 (+ 794us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.189632 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":37}
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
I20260928 14:40:27.190635 28740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:27.193308 28589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e: Bootstrap starting.
I20260928 14:40:27.194403 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.183936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59170)
0928 14:40:27.184139 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.184155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:27.184233 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:27.184635 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:27.184638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:27.184808 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:27.184885 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.184890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.186614 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.186618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.187526 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.187537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.188186 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.188190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.188275 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.188759 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:27.188776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:27.190792 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:27.192883 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.192886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.192888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.193093 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.193381 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.193384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.193387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.193479 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.193873 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:27.193876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:27.193979 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 14:40:27.194083 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 14:40:27.194230 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 14:40:27.194422 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.184017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.6.23.252:43703)
0928 14:40:27.184375 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.184389 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.184399 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:40:27.184425 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.184427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.184457 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.184623 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.184995 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.186475 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.187664 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.188059 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.188891 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.190929 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.190933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.190935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.190961 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.192737 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.193195 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.193199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.193201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.193288 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.193585 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.193588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.193590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.193773 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.193878 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.194146 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.194255 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20260928 14:40:27.196085 28589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:27.197031 28589 log.cc:824] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:27.197629 28664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6: Bootstrap starting.
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606427, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
I20260928 14:40:27.200228 28664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:27.201136 28664 log.cc:824] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:40:27 dist-test-slave-06lh krb5kdc[28508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:40:27.203627 28664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6: No bootstrap required, opened a new log
I20260928 14:40:27.203673 28589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e: No bootstrap required, opened a new log
I20260928 14:40:27.204407 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.191071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:43703 (local address 127.0.0.1:59182)
0928 14:40:27.191173 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:27.191186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:27.191284 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:27.195703 (+ 4419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:27.195705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:27.195878 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:27.195979 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.195987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.197006 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.197009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.197763 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:27.197773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.198347 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.198351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:27.198421 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.198933 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:27.198945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:27.200315 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:27.202874 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.202877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.202879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.203105 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.203431 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:27.203434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:27.203436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:27.203515 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:27.203932 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:27.203937 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:27.204027 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 14:40:27.204107 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:40:27.204245 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 14:40:27.205034 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.195372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.6.23.252:43703)
0928 14:40:27.195482 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.195486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.195499 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:40:27.195533 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.195535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.195585 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.196075 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.196151 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.196909 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.197875 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.198230 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.198860 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.200448 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.200451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.200454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.200475 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.202744 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.203213 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.203216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.203218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.203336 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.203610 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.203612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.203614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.203832 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.204697 (+ 865us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.204779 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.204894 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:40:27.205513 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.191022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48730 (local address 127.6.23.253:43153)
0928 14:40:27.191200 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.191206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.195619 (+ 4413us) server_negotiation.cc:407] Connection header received
0928 14:40:27.195832 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.195835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.195881 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.195928 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.196804 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.197537 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.198643 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.198953 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.199058 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.201842 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.201846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.201849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.201872 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.203534 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.204030 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.204034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.204036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.204128 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.204460 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.204464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.204465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.204688 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.204851 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.205241 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.205380 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260928 14:40:27.207924 28664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } }
I20260928 14:40:27.208176 28664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1015335f40f94f359ede0ce18531aeb6, State: Initialized, Role: FOLLOWER
I20260928 14:40:27.208329 28740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26: Bootstrap starting.
I20260928 14:40:27.208545 28664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } }
I20260928 14:40:27.209306 28748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.209460 28748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:27.209794 28664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:27.211009 28740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:27.212886 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:40:27.213290 28740 log.cc:824] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:27.214334 28563 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f2e6726b8995415694bf29d99062287e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37006
W20260928 14:40:27.215153 28759 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:27.215226 28759 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:27.213505 28596 master_runner.cc:186] Error getting master registration for 127.6.23.254:45981: OK, instance_id { permanent_uuid: "f2e6726b8995415694bf29d99062287e" instance_seqno: 1790606426706169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:40:27.216933 28596 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:40:27.216871 28589 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } }
I20260928 14:40:27.217128 28589 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e6726b8995415694bf29d99062287e, State: Initialized, Role: FOLLOWER
I20260928 14:40:27.217448 28589 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } }
I20260928 14:40:27.217979 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.218132 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:27.218788 28589 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:27.218847 28563 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:40:27.218986 28563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:27.220731 28563 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } }
I20260928 14:40:27.221172 28740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26: No bootstrap required, opened a new log
I20260928 14:40:27.221827 28563 leader_election.cc:288] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [CANDIDATE]: Term 1 election: Requested vote from peers 1015335f40f94f359ede0ce18531aeb6 (127.6.23.253:43153), 2983e92ed2164ca6b5f3a63a400b6a26 (127.6.23.252:43703)
I20260928 14:40:27.222574 28638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2e6726b8995415694bf29d99062287e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1015335f40f94f359ede0ce18531aeb6"
I20260928 14:40:27.222872 28638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:27.224186 28638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2e6726b8995415694bf29d99062287e in term 1.
I20260928 14:40:27.224651 28539 leader_election.cc:302] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1015335f40f94f359ede0ce18531aeb6, f2e6726b8995415694bf29d99062287e; no voters:
I20260928 14:40:27.225031 28762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:27.225272 28762 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [term 1 LEADER]: Becoming Leader. State: Replica: f2e6726b8995415694bf29d99062287e, State: Running, Role: LEADER
I20260928 14:40:27.225664 28762 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } }
I20260928 14:40:27.227835 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2e6726b8995415694bf29d99062287e. Latest consensus state: current_term: 1 leader_uuid: "f2e6726b8995415694bf29d99062287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.227954 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:27.228507 28774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:27.231449 28740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } }
I20260928 14:40:27.231778 28740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2983e92ed2164ca6b5f3a63a400b6a26, State: Initialized, Role: FOLLOWER
I20260928 14:40:27.232182 28740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } }
I20260928 14:40:27.233770 28740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:27.233819 28714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f2e6726b8995415694bf29d99062287e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2983e92ed2164ca6b5f3a63a400b6a26"
I20260928 14:40:27.234076 28714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:27.236109 28714 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2e6726b8995415694bf29d99062287e in term 1.
I20260928 14:40:27.237071 28776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.237197 28670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.224280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.6.23.253:43153)
0928 14:40:27.224442 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.224449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.226021 (+ 1572us) server_negotiation.cc:407] Connection header received
0928 14:40:27.226166 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.226169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.226226 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.226323 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.227355 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.228101 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.229993 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.230344 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.230453 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.232702 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.232706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.232727 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.232759 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.234899 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.235577 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.235581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.235585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.235678 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.236005 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.236008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.236010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.236286 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.236419 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.236887 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.237025 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 14:40:27.237201 28776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:27.238178 28774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 14:40:27.242148 28795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:27.242236 28795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:27.251410 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.224815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37012 (local address 127.6.23.254:45981)
0928 14:40:27.224934 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.224939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.241688 (+ 16749us) server_negotiation.cc:407] Connection header received
0928 14:40:27.241806 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.241809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.241859 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.241909 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.242988 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.243779 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.244886 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.245198 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.245289 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.247492 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.247496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.247500 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.247530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.249318 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.249869 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.249873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.249875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.249969 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.250300 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.250304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.250306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.250547 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.250673 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.251064 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.251195 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:40:27.261029 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:27.225336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.6.23.252:43703)
0928 14:40:27.225450 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.225455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.251736 (+ 26281us) server_negotiation.cc:407] Connection header received
0928 14:40:27.251781 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.251783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.251828 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.251901 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.252904 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.253707 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.254882 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:27.255255 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:27.255334 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:27.257155 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:27.257159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:27.257162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:27.257189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:27.258942 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.259494 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.259499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.259501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.259592 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:27.259902 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:27.259906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:27.259908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:27.260125 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:27.260223 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:27.260646 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 14:40:27.260805 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:40:27.273515 28714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [term 1 FOLLOWER]: Refusing update from remote peer f2e6726b8995415694bf29d99062287e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:27.273576 28638 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [term 1 FOLLOWER]: Refusing update from remote peer f2e6726b8995415694bf29d99062287e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:27.274313 28770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:27.274504 28762 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:27.278560 28776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2e6726b8995415694bf29d99062287e. Latest consensus state: current_term: 1 leader_uuid: "f2e6726b8995415694bf29d99062287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.278576 28748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2e6726b8995415694bf29d99062287e. Latest consensus state: current_term: 1 leader_uuid: "f2e6726b8995415694bf29d99062287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.278697 28748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:27.278698 28776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:27.284240 28762 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2e6726b8995415694bf29d99062287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.284430 28762 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:27.284821 28798 mvcc.cc:205] Tried to move back new op lower bound from 7334323926092374016 to 7334323925920763904. Current Snapshot: MvccSnapshot[applied={T|T < 7334323926092374016}]
I20260928 14:40:27.286566 28748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2e6726b8995415694bf29d99062287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.286688 28748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:27.287843 28770 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2e6726b8995415694bf29d99062287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.288003 28770 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:27.288856 28799 mvcc.cc:205] Tried to move back new op lower bound from 7334323926092374016 to 7334323925920763904. Current Snapshot: MvccSnapshot[applied={T|T < 7334323926092374016}]
I20260928 14:40:27.289033 28776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2e6726b8995415694bf29d99062287e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e6726b8995415694bf29d99062287e" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 45981 } } peers { permanent_uuid: "1015335f40f94f359ede0ce18531aeb6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 43153 } } peers { permanent_uuid: "2983e92ed2164ca6b5f3a63a400b6a26" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 43703 } } }
I20260928 14:40:27.289160 28776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:27.291337 28800 mvcc.cc:205] Tried to move back new op lower bound from 7334323926092374016 to 7334323925920763904. Current Snapshot: MvccSnapshot[applied={T|T < 7334323926092374016}]
I20260928 14:40:27.299729 28774 catalog_manager.cc:1347] Generated new cluster ID: b1633c2cd97248f1a957b552ebab962f
I20260928 14:40:27.299836 28774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:27.318638 28774 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:27.320998 28774 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:27.335739 28774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2e6726b8995415694bf29d99062287e: Generated new TSK 0
I20260928 14:40:27.336740 28774 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:40:27.372426 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.369061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48738 (local address 127.6.23.253:43153)
0928 14:40:27.369216 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.369221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.371182 (+ 1961us) server_negotiation.cc:407] Connection header received
0928 14:40:27.371429 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.371433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.371491 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.371546 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.372270 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48738: BlockingRecv error: recv got EOF from 127.0.0.1:48738 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 14:40:27.373975 28745 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.369915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59210 (local address 127.6.23.252:43703)
0928 14:40:27.370059 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.370064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.373003 (+ 2939us) server_negotiation.cc:407] Connection header received
0928 14:40:27.373186 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.373190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.373244 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.373299 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.373846 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59210: BlockingRecv error: recv got EOF from 127.0.0.1:59210 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 14:40:27.375665 28593 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.370522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.6.23.254:45981)
0928 14:40:27.370642 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.370647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.374383 (+ 3736us) server_negotiation.cc:407] Connection header received
0928 14:40:27.374837 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.374842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.374899 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.374953 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.375540 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37014: BlockingRecv error: recv got EOF from 127.0.0.1:37014 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:40:27.380661 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.378805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.6.23.253:43153)
0928 14:40:27.378940 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.378944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.379509 (+ 565us) server_negotiation.cc:407] Connection header received
0928 14:40:27.379802 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.379807 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.379891 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.379948 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.380542 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48746: BlockingRecv error: recv got EOF from 127.0.0.1:48746 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 14:40:27.386185 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.384444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.6.23.253:43153)
0928 14:40:27.384555 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.384559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.385155 (+ 596us) server_negotiation.cc:407] Connection header received
0928 14:40:27.385372 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.385375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.385435 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.385491 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.386073 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48760: BlockingRecv error: recv got EOF from 127.0.0.1:48760 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:40:27.392139 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.390373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.6.23.253:43153)
0928 14:40:27.390484 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.390488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.391089 (+ 601us) server_negotiation.cc:407] Connection header received
0928 14:40:27.391325 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.391328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.391385 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.391443 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.392022 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48776: BlockingRecv error: recv got EOF from 127.0.0.1:48776 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 14:40:27.400327 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.398412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.6.23.253:43153)
0928 14:40:27.398568 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.398573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.399208 (+ 635us) server_negotiation.cc:407] Connection header received
0928 14:40:27.399490 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.399494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.399554 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.399612 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.400208 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48790: BlockingRecv error: recv got EOF from 127.0.0.1:48790 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 14:40:27.412812 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.410844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.6.23.253:43153)
0928 14:40:27.410985 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.410990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.411675 (+ 685us) server_negotiation.cc:407] Connection header received
0928 14:40:27.411898 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.411902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.411977 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.412032 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.412661 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48802: BlockingRecv error: recv got EOF from 127.0.0.1:48802 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 14:40:27.432605 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.431269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.6.23.253:43153)
0928 14:40:27.431404 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.431410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.431676 (+ 266us) server_negotiation.cc:407] Connection header received
0928 14:40:27.431857 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.431860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.431911 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.432004 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.432481 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48814: BlockingRecv error: recv got EOF from 127.0.0.1:48814 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 14:40:27.467945 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.466588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.6.23.253:43153)
0928 14:40:27.466724 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.466731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.466936 (+ 205us) server_negotiation.cc:407] Connection header received
0928 14:40:27.467127 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.467137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.467189 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.467308 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.467799 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48816: BlockingRecv error: recv got EOF from 127.0.0.1:48816 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 14:40:27.535782 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.534315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.6.23.253:43153)
0928 14:40:27.534566 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.534573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.534768 (+ 195us) server_negotiation.cc:407] Connection header received
0928 14:40:27.534959 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.534963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.535029 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.535125 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.535658 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48818: BlockingRecv error: recv got EOF from 127.0.0.1:48818 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260928 14:40:27.667656 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.666377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.6.23.253:43153)
0928 14:40:27.666546 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.666551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.666684 (+ 133us) server_negotiation.cc:407] Connection header received
0928 14:40:27.666875 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.666878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.666936 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.667029 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.667535 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48830: BlockingRecv error: recv got EOF from 127.0.0.1:48830 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 14:40:27.927013 28670 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:27.925911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.6.23.253:43153)
0928 14:40:27.926058 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:27.926062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:27.926174 (+ 112us) server_negotiation.cc:407] Connection header received
0928 14:40:27.926341 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:27.926344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:27.926398 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:27.926482 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:27.926912 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48840: BlockingRecv error: recv got EOF from 127.0.0.1:48840 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:40:28.216619 28759 catalog_manager.cc:1259] Loaded cluster ID: b1633c2cd97248f1a957b552ebab962f
I20260928 14:40:28.216711 28759 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6: loading cluster ID for follower catalog manager: success
I20260928 14:40:28.218418 28759 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6: acquiring CA information for follower catalog manager: success
I20260928 14:40:28.219700 28759 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1015335f40f94f359ede0ce18531aeb6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:40:28.243611 28795 catalog_manager.cc:1259] Loaded cluster ID: b1633c2cd97248f1a957b552ebab962f
I20260928 14:40:28.243745 28795 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26: loading cluster ID for follower catalog manager: success
I20260928 14:40:28.245433 28795 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26: acquiring CA information for follower catalog manager: success
I20260928 14:40:28.246698 28795 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2983e92ed2164ca6b5f3a63a400b6a26: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:40:28.445591 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:28.441276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.6.23.253:43153)
0928 14:40:28.441543 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:28.441548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:28.441671 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:40:28.441854 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:28.441857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:28.441991 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:28.442139 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:28.442449 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.442989 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.443996 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.444247 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.444361 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:28.444706 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:28.445173 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:28.445316 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:28.445356 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 14:40:28.445419 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260928 14:40:28.453130 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:28.449127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.6.23.252:43703)
0928 14:40:28.449379 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:28.449383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:28.449414 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:40:28.449587 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:28.449590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:28.449722 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:28.449822 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:28.450090 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.450628 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.451609 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.451844 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.451925 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:28.452190 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:28.452653 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:28.452747 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:28.452861 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 14:40:28.452916 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":107,"threads_started":1}
I20260928 14:40:28.462986 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:28.457795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53690 (local address 127.6.23.253:43153)
0928 14:40:28.457930 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:28.457936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:28.457952 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:40:28.458008 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:28.458012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:28.458078 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:28.458393 (+ 315us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:28.458646 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.459540 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.461109 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.461503 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.461613 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:28.462452 (+ 839us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:28.462571 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:28.462645 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:28.462751 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 14:40:28.462828 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 14:40:28.463646 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:28.457960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.6.23.254:45981)
0928 14:40:28.458270 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:28.458274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:28.458287 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:40:28.458403 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:28.458406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:28.458580 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:28.458699 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:28.459223 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.460090 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.461412 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.461859 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.461955 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:28.462429 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:28.462819 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:28.462903 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:28.463406 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 14:40:28.463467 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":92,"threads_started":1}
I20260928 14:40:28.465322 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:28.457629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42882 (local address 127.6.23.252:43703)
0928 14:40:28.457819 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:28.457826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:28.458637 (+ 811us) server_negotiation.cc:407] Connection header received
0928 14:40:28.458689 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:28.458692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:28.458743 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:28.458808 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:28.461852 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.462514 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.463946 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:28.464338 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:28.464463 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:28.464712 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:28.464800 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:28.464887 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:28.465081 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 14:40:28.465158 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 14:40:28.474864 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28521
I20260928 14:40:28.481385 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28596
I20260928 14:40:28.487389 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28672
2026-09-28T14:40:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3147 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:40:28 dist-test-slave-06lh krb5kdc[28888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:40:28 dist-test-slave-06lh krb5kdc[28888](info): set up 2 sockets
Sep 28 14:40:28 dist-test-slave-06lh krb5kdc[28888](info): commencing operation
krb5kdc: starting...
W20260928 14:40:29.544010 6239 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/test-user.keytab.
Sep 28 14:40:29 dist-test-slave-06lh krb5kdc[28888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:40:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:40:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.23.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
Entry for principal kudu/127.6.23.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab.
I20260928 14:40:29.672936 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35703
--webserver_interface=127.6.23.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:35703,127.6.23.253:33727,127.6.23.252:37353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:40:29.787401 28901 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:40:29.787775 28901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:29.787842 28901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:29.787922 28901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:29.792225 28901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:29.792323 28901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:29.792371 28901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:29.792430 28901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:29.792469 28901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:29.798111 28901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:35703,127.6.23.253:33727,127.6.23.252:37353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.254.keytab
--principal=kudu/127.6.23.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.254:35703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.23.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:29.799379 28901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:29.800467 28901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:29.809302 28906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:29.809260 28907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:29.809274 28909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:29.809418 28901 server_base.cc:1034] running on GCE node
I20260928 14:40:29.810146 28901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:29.811378 28901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:29.812549 28901 hybrid_clock.cc:648] HybridClock initialized: now 1790606429812533 us; error 36 us; skew 500 ppm
Sep 28 14:40:29 dist-test-slave-06lh krb5kdc[28888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:29.815459 28901 init.cc:377] Logged in from keytab as kudu/127.6.23.254@KRBTEST.COM (short username kudu)
I20260928 14:40:29.816409 28901 webserver.cc:460] Webserver started at http://127.6.23.254:44295/ using document root <none> and password file <none>
I20260928 14:40:29.816983 28901 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:29.817077 28901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:29.817318 28901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:29.819479 28901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/data/instance:
uuid: "6761660fc05e4be7abbeb01c373b01f6"
format_stamp: "Formatted at 2026-09-28 14:40:29 on dist-test-slave-06lh"
server_key: "554529f2d574dd93304dad1e8e3f6137"
server_key_iv: "12b1d372cd6b88adf3fac4cdca68bbc0"
server_key_version: "encryptionkey@0"
I20260928 14:40:29.820081 28901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/wal/instance:
uuid: "6761660fc05e4be7abbeb01c373b01f6"
format_stamp: "Formatted at 2026-09-28 14:40:29 on dist-test-slave-06lh"
server_key: "554529f2d574dd93304dad1e8e3f6137"
server_key_iv: "12b1d372cd6b88adf3fac4cdca68bbc0"
server_key_version: "encryptionkey@0"
I20260928 14:40:29.823858 28901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:40:29.826326 28916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:29.827666 28901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 14:40:29.827813 28901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/wal
uuid: "6761660fc05e4be7abbeb01c373b01f6"
format_stamp: "Formatted at 2026-09-28 14:40:29 on dist-test-slave-06lh"
server_key: "554529f2d574dd93304dad1e8e3f6137"
server_key_iv: "12b1d372cd6b88adf3fac4cdca68bbc0"
server_key_version: "encryptionkey@0"
I20260928 14:40:29.827934 28901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:29.844493 28901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:29.848024 28901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:29.848250 28901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:29.855623 28901 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.254:35703
I20260928 14:40:29.855919 28968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.254:35703 every 8 connection(s)
I20260928 14:40:29.856676 28901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/data/info.pb
I20260928 14:40:29.859489 28969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:29.859674 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28901
I20260928 14:40:29.859800 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-0/wal/instance
I20260928 14:40:29.860113 6239 external_mini_cluster.cc:1428] Setting key 7f6f03d8ff5ef7b91a678734a4154b1d
I20260928 14:40:29.863453 28969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.23.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.253@KRBTEST.COM" created.
Sep 28 14:40:29 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:29.880038 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:29.865905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.6.23.254:35703)
0928 14:40:29.866357 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:29.866379 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:29.866653 (+ 274us) server_negotiation.cc:407] Connection header received
0928 14:40:29.867803 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:29.867808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:29.868001 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:29.868227 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:29.869119 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:29.870056 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:29.871180 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:29.871625 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:29.871822 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:29.874948 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:29.874954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:29.874966 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:29.874988 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:29.876689 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:29.877358 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:29.877363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:29.877364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:29.877467 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:29.877847 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:29.877850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:29.877851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:29.878245 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:29.878366 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:29.878884 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 14:40:29.879147 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
I20260928 14:40:29.880041 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:29.865906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35703 (local address 127.0.0.1:56756)
0928 14:40:29.866501 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:29.866538 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:29.867490 (+ 952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:29.868222 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:29.868227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:29.868656 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:29.868943 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:29.868964 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:29.870212 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:29.870215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:29.871053 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:29.871064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:29.871814 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:29.871818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:29.871930 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:29.872669 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:29.872696 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:29.874770 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:29.876825 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:29.876839 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:29.876852 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:29.877232 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:29.877565 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:29.877569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:29.877572 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:29.877741 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:29.878390 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:29.878398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:29.878618 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 14:40:29.878834 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 14:40:29.879278 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":80,"thread_start_us":163,"threads_started":1}
I20260928 14:40:29.882570 28969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:29.883426 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:29.883064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59778)
0928 14:40:29.883286 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:29.883364 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260928 14:40:29.883652 28920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
I20260928 14:40:29.884467 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:29.884272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59794)
0928 14:40:29.884373 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:29.884416 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:40:29.884711 28969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:33727: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab.
Entry for principal kudu/127.6.23.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab.
I20260928 14:40:29.890470 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.253:33727
--webserver_interface=127.6.23.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab
--principal=kudu/127.6.23.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:35703,127.6.23.253:33727,127.6.23.252:37353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:40:29.934051 28969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } attempt: 1
I20260928 14:40:29.935041 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:29.934668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59806)
0928 14:40:29.934819 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:29.934937 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:40:29.935947 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:29.935719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59822)
0928 14:40:29.935832 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:29.935880 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:40:29.936154 28969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:33727: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
I20260928 14:40:30.004346 28969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } attempt: 2
W20260928 14:40:30.004354 28976 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:40:30.004741 28976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:30.004808 28976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:30.004889 28976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:40:30.005296 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.005005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59838)
0928 14:40:30.005153 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.005217 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 14:40:30.006073 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.005850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59846)
0928 14:40:30.005961 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.006015 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 14:40:30.006237 28969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.253:33727: Network error: Client connection negotiation failed: client connection to 127.6.23.253:33727: connect: Connection refused (error 111)
W20260928 14:40:30.009518 28976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:30.009606 28976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:30.009656 28976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:30.009694 28976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:30.009752 28976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:30.015522 28976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:35703,127.6.23.253:33727,127.6.23.252:37353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.253.keytab
--principal=kudu/127.6.23.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.253:33727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.23.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:30.016824 28976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:30.017949 28976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:30.027366 28984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:30.027366 28981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:30.027613 28982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:30.027698 28976 server_base.cc:1034] running on GCE node
I20260928 14:40:30.028358 28976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:30.029543 28976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:30.030712 28976 hybrid_clock.cc:648] HybridClock initialized: now 1790606430030695 us; error 30 us; skew 500 ppm
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:30.033835 28976 init.cc:377] Logged in from keytab as kudu/127.6.23.253@KRBTEST.COM (short username kudu)
I20260928 14:40:30.034794 28976 webserver.cc:460] Webserver started at http://127.6.23.253:44319/ using document root <none> and password file <none>
I20260928 14:40:30.035436 28976 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:30.035523 28976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:30.035756 28976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:30.037849 28976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/data/instance:
uuid: "2d115b3f905c4323a31c45bc210d60d6"
format_stamp: "Formatted at 2026-09-28 14:40:30 on dist-test-slave-06lh"
server_key: "3c3eac6a589d965c98f9affd01eeeb61"
server_key_iv: "a04ebbe8c23267cad4b9a3edebfa80b2"
server_key_version: "encryptionkey@0"
I20260928 14:40:30.038424 28976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/wal/instance:
uuid: "2d115b3f905c4323a31c45bc210d60d6"
format_stamp: "Formatted at 2026-09-28 14:40:30 on dist-test-slave-06lh"
server_key: "3c3eac6a589d965c98f9affd01eeeb61"
server_key_iv: "a04ebbe8c23267cad4b9a3edebfa80b2"
server_key_version: "encryptionkey@0"
I20260928 14:40:30.042292 28976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:40:30.044947 28991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:30.046300 28976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 14:40:30.046481 28976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/wal
uuid: "2d115b3f905c4323a31c45bc210d60d6"
format_stamp: "Formatted at 2026-09-28 14:40:30 on dist-test-slave-06lh"
server_key: "3c3eac6a589d965c98f9affd01eeeb61"
server_key_iv: "a04ebbe8c23267cad4b9a3edebfa80b2"
server_key_version: "encryptionkey@0"
I20260928 14:40:30.046607 28976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:30.064373 28976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:30.067941 28976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:30.068159 28976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:30.075774 28976 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.253:33727
I20260928 14:40:30.076027 29043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.253:33727 every 8 connection(s)
I20260928 14:40:30.076840 28976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/data/info.pb
I20260928 14:40:30.077597 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 28976
I20260928 14:40:30.077735 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-1/wal/instance
I20260928 14:40:30.078029 6239 external_mini_cluster.cc:1428] Setting key 1614864072b7bc76b2d385d72bc4c14b
I20260928 14:40:30.080204 29044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:30.085064 29044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.23.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.23.252@KRBTEST.COM" created.
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:30.097394 28969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } attempt: 3
I20260928 14:40:30.102751 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.087597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.6.23.254:35703)
0928 14:40:30.087761 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.087766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.088914 (+ 1148us) server_negotiation.cc:407] Connection header received
0928 14:40:30.089923 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.089926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.089978 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.090053 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.091406 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.092248 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.094013 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.094456 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.094587 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.097582 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.097587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.097589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.097621 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.100209 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.100770 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.100773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.100775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.100865 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.101445 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.101449 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.101451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.101662 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.101739 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.102422 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.102545 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":58}
I20260928 14:40:30.103713 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.088046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35703 (local address 127.0.0.1:56768)
0928 14:40:30.088735 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.088796 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:30.089730 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:30.090243 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:30.090250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:30.090706 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:30.091196 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.091210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.092426 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.092432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.093875 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.093887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.094618 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.094621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.094779 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.095469 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:30.095486 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:30.097363 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:30.100334 (+ 2971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.100340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.100354 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.100630 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.100984 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.100987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.100988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.101262 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.101773 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:30.101780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:30.102096 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 14:40:30.102331 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 14:40:30.102621 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":120,"threads_started":1}
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:40:30.105370 29044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.23.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab.
Entry for principal kudu/127.6.23.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab.
I20260928 14:40:30.109637 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.097912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59854)
0928 14:40:30.098073 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.098091 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:30.098172 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:30.099038 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:30.099041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:30.099295 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:30.099438 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.099448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.101478 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.101482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.102207 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.102218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.103050 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.103054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.103147 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.103727 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:30.103747 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:30.105287 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:30.107799 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.107802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.107804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.108092 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.108447 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.108453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.108458 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.108556 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.109139 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:30.109142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:30.109247 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 14:40:30.109346 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:40:30.109485 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:40:30.110138 6239 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
/tmp/dist-test-taskpOhn9p/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.252:37353
--webserver_interface=127.6.23.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab
--principal=kudu/127.6.23.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.23.212:37597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.23.254:35703,127.6.23.253:33727,127.6.23.252:37353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:40:30.110278 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.098353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.6.23.253:33727)
0928 14:40:30.098664 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.098678 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.098700 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:40:30.098737 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.098742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.098908 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.099060 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.100317 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.101296 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.102351 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.102914 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.103404 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.105418 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.105427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.105440 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.105466 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.107689 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.108236 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.108241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.108242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.108330 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.108658 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.108661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.108663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.109012 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.109867 (+ 855us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.110003 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.110145 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":132,"threads_started":1}
I20260928 14:40:30.112159 28969 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:40:30.112859 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.112617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40344)
0928 14:40:30.112738 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.112794 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 14:40:30.113647 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.113318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40360)
0928 14:40:30.113508 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.113575 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260928 14:40:30.113837 28969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:37353: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
I20260928 14:40:30.116786 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.106009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.6.23.253:33727)
0928 14:40:30.106417 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.106427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.106440 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:40:30.106473 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.106478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.106550 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.106644 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.107042 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.107914 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.108824 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.109210 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.109301 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.113117 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.113123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.113128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.113158 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.114924 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.115440 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.115442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.115443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.115520 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.115831 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.115833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.115834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.116006 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.116081 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.116411 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.116641 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":322,"thread_start_us":167,"threads_started":1}
I20260928 14:40:30.116808 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.105969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59870)
0928 14:40:30.106155 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.106173 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:30.106252 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:30.106685 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:30.106687 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:30.106878 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:30.106953 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.106958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.108018 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.108025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.108706 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.108716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.109342 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.109349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.109443 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.109994 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:30.110010 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:30.112821 (+ 2811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:30.115016 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.115022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.115027 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.115339 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.115616 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.115621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.115625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.115728 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.116110 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:30.116116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:30.116237 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 14:40:30.116365 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 14:40:30.116558 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 14:40:30.117621 29044 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:30.118396 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.118059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40364)
0928 14:40:30.118207 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.118311 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 14:40:30.118604 28995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
I20260928 14:40:30.119638 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.119361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40370)
0928 14:40:30.119489 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.119549 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:40:30.119797 29044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:37353: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
I20260928 14:40:30.145056 28969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } attempt: 1
I20260928 14:40:30.146021 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.145702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40372)
0928 14:40:30.145859 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.145931 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 14:40:30.146880 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.146658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40374)
0928 14:40:30.146770 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.146819 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 14:40:30.147058 28969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:37353: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
I20260928 14:40:30.169109 29044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } attempt: 1
I20260928 14:40:30.170109 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.169752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40386)
0928 14:40:30.169934 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.170005 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:40:30.170861 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.170633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40394)
0928 14:40:30.170746 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.170799 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 14:40:30.171018 29044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:37353: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
I20260928 14:40:30.222277 28969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } attempt: 2
I20260928 14:40:30.223285 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.222921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40398)
0928 14:40:30.223076 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.223156 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 14:40:30.224031 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.223800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40410)
0928 14:40:30.223917 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.223972 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:40:30.224169 28969 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:37353: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
W20260928 14:40:30.228300 29051 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:40:30.228674 29051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:40:30.228740 29051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:40:30.228822 29051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:40:30.233194 29051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:40:30.233284 29051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:40:30.233320 29051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:40:30.233358 29051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:40:30.233397 29051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:40:30.239114 29051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.23.212:37597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.23.254:35703,127.6.23.253:33727,127.6.23.252:37353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/krb5kdc/kudu_127.6.23.252.keytab
--principal=kudu/127.6.23.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.23.252:37353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.23.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:40:30.239313 29044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } attempt: 2
I20260928 14:40:30.240288 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.239928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40422)
0928 14:40:30.240118 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.240191 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 14:40:30.240592 29051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:40:30.241123 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.240880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40436)
0928 14:40:30.240997 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.241049 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 14:40:30.241287 29044 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.23.252:37353: Network error: Client connection negotiation failed: client connection to 127.6.23.252:37353: connect: Connection refused (error 111)
I20260928 14:40:30.241882 29051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:40:30.251098 29058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:40:30.251122 29060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:30.251150 29051 server_base.cc:1034] running on GCE node
W20260928 14:40:30.251286 29057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:40:30.251946 29051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:40:30.253167 29051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:40:30.254338 29051 hybrid_clock.cc:648] HybridClock initialized: now 1790606430254301 us; error 49 us; skew 500 ppm
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:40:30.257299 29051 init.cc:377] Logged in from keytab as kudu/127.6.23.252@KRBTEST.COM (short username kudu)
I20260928 14:40:30.258338 29051 webserver.cc:460] Webserver started at http://127.6.23.252:42079/ using document root <none> and password file <none>
I20260928 14:40:30.259017 29051 fs_manager.cc:359] Metadata directory not provided
I20260928 14:40:30.259091 29051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:40:30.259353 29051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:40:30.261521 29051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/data/instance:
uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5"
format_stamp: "Formatted at 2026-09-28 14:40:30 on dist-test-slave-06lh"
server_key: "37398e12c2c3e72d7521a34c37a4080f"
server_key_iv: "b4e02b31eba280b8bb3fbeb017a44b99"
server_key_version: "encryptionkey@0"
I20260928 14:40:30.262137 29051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/wal/instance:
uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5"
format_stamp: "Formatted at 2026-09-28 14:40:30 on dist-test-slave-06lh"
server_key: "37398e12c2c3e72d7521a34c37a4080f"
server_key_iv: "b4e02b31eba280b8bb3fbeb017a44b99"
server_key_version: "encryptionkey@0"
I20260928 14:40:30.266130 29051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:40:30.268862 29067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:40:30.270226 29051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:40:30.270401 29051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/data,/tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/wal
uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5"
format_stamp: "Formatted at 2026-09-28 14:40:30 on dist-test-slave-06lh"
server_key: "37398e12c2c3e72d7521a34c37a4080f"
server_key_iv: "b4e02b31eba280b8bb3fbeb017a44b99"
server_key_version: "encryptionkey@0"
I20260928 14:40:30.270535 29051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:40:30.284740 29051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:40:30.288522 29051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:40:30.288755 29051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:40:30.296558 29051 rpc_server.cc:307] RPC server started. Bound to: 127.6.23.252:37353
I20260928 14:40:30.296867 29119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.23.252:37353 every 8 connection(s)
I20260928 14:40:30.297731 29051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/data/info.pb
I20260928 14:40:30.298355 6239 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu as pid 29051
I20260928 14:40:30.298519 6239 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskpOhn9p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606142807864-6239-0/minicluster-data/master-2/wal/instance
I20260928 14:40:30.298842 6239 external_mini_cluster.cc:1428] Setting key 1d13a438e8e9cd075f0b89661d8e2225
I20260928 14:40:30.301208 29120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:40:30.305182 29120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:30.314218 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.300348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.6.23.252:37353)
0928 14:40:30.300818 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.300833 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.300870 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:40:30.301628 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.301649 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.302007 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.302371 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.303211 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.304308 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.305241 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.305794 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.305961 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.308946 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.308953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.308971 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.308997 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.311036 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.311513 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.311518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.311525 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.311705 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.311985 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.311989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.311991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.312521 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.312671 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.313061 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.313434 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":155,"threads_started":1}
I20260928 14:40:30.318091 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.307140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.6.23.254:35703)
0928 14:40:30.307381 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.307387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.307700 (+ 313us) server_negotiation.cc:407] Connection header received
0928 14:40:30.307844 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.307849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.307906 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.307991 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.308730 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.309367 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.310738 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.310949 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.311032 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.313560 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.313563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.313565 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.313590 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.315832 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.316566 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.316569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.316603 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.316739 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.317099 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.317102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.317104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.317332 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.317463 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.317838 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.317970 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 14:40:30.318243 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.307030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.254:35703 (local address 127.0.0.1:56784)
0928 14:40:30.307551 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.307602 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:30.307730 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:30.308050 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:30.308054 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:30.308465 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:30.308603 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.308608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.309504 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.309507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.310594 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.310605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.311063 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.311069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.311207 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.311739 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:30.311767 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:30.313411 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:30.315955 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.315967 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.315973 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.316249 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.316896 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.316902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.316907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.317014 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.317452 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:30.317459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:30.317582 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 14:40:30.317767 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 14:40:30.317988 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":236,"threads_started":1}
I20260928 14:40:30.319600 29120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.254@KRBTEST.COM
I20260928 14:40:30.323299 28969 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } attempt: 3
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:40:30.327386 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.316780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.6.23.254:35703)
0928 14:40:30.317341 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.317369 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.317398 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:40:30.317457 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.317460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.317519 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.317774 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.318590 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.319461 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.320417 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.320814 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.320923 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.323617 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.323624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.323630 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.323662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.325556 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.326089 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.326092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.326094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.326180 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.326466 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.326471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.326472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.326662 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.326877 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.327014 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.327148 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606429, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.254@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
I20260928 14:40:30.329315 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.320257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.6.23.253:33727)
0928 14:40:30.320414 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.320419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.320434 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:40:30.320549 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.320552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.320603 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.320687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.321113 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.321639 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.322408 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.322672 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.322789 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.325224 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.325228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.325231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.325268 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.326992 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.327532 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.327536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.327537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.327634 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.328264 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.328268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.328270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.328523 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.328665 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.328975 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.329136 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:40:30.329834 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.320112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.253:33727 (local address 127.0.0.1:59884)
0928 14:40:30.320390 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.320402 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:30.320464 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:30.320737 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:30.320739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:30.320885 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:30.320979 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.320985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.321760 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.321763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.322292 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.322300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.322793 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.322796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.322899 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.323475 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:30.323498 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:30.325056 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:30.327120 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.327123 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.327125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.327430 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.328052 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.328055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.328057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.328156 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.328657 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:30.328660 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:30.328763 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 14:40:30.329501 (+ 738us) client_negotiation.cc:240] Negotiation successful
0928 14:40:30.329669 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260928 14:40:30.330636 29120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:40:30.332487 29044 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } attempt: 3
I20260928 14:40:30.334584 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.323779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40454)
0928 14:40:30.323899 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.323916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:30.324001 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:30.324371 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:30.324374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:30.324592 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:30.324730 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.324738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.325787 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.325790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.326697 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.326711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.327426 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.327429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.327532 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.327968 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:30.327986 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:30.329500 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:30.332893 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.332896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.332898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.333174 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.333503 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.333506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.333508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.333598 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.334049 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:30.334053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:30.334158 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 14:40:30.334244 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 14:40:30.334412 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 14:40:30.335193 29123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.323972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.6.23.252:37353)
0928 14:40:30.324105 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.324111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.324124 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:40:30.324158 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.324162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.324203 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.324359 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.324890 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.325633 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.326873 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.327288 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.327756 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.329662 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.329667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.329669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.329697 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.332764 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.333301 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.333304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.333306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.333401 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.333720 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.333726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.333728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.333938 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.334784 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.334864 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.335018 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.252@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
I20260928 14:40:30.339048 28969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6: Bootstrap starting.
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606430, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.23.253@KRBTEST.COM for kudu/127.6.23.252@KRBTEST.COM
I20260928 14:40:30.341269 29124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.331324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40466)
0928 14:40:30.331539 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.331557 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:30.331644 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:30.331948 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:30.331950 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:30.332145 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:30.332207 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.332212 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.333824 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.333827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.334660 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.334672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.335493 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.335497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.335578 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.336115 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:30.336129 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:30.337451 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:30.339595 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.339601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.339604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.339885 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.340117 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.340122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.340124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.340206 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.340688 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:30.340691 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:30.340778 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 14:40:30.340893 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:40:30.341036 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":3}
I20260928 14:40:30.341490 29126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.331334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.6.23.252:37353)
0928 14:40:30.331755 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.331765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.331776 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:40:30.331810 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.331811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.331842 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.331939 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.332350 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.333718 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.334787 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.335336 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.335789 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.337589 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.337593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.337595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.337617 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.339481 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.339971 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.339972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.339974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.340028 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.340301 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.340304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.340306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.340552 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.340649 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.340938 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.341330 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":144,"threads_started":1}
I20260928 14:40:30.342338 28969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:30.343328 28969 log.cc:824] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:30.344035 29047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.332928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.23.252:37353 (local address 127.0.0.1:40472)
0928 14:40:30.333220 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:40:30.333237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:40:30.333321 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:40:30.333569 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:40:30.333572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:40:30.333756 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:40:30.333868 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.333875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.336391 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.336395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.337140 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:40:30.337152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.337774 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.337778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:40:30.337864 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.338348 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:40:30.338362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:40:30.339971 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:40:30.342300 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.342303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.342305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.342568 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.342884 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:40:30.342887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:40:30.342889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:40:30.342979 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:40:30.343490 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:40:30.343493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:40:30.343595 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 14:40:30.343682 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:40:30.343838 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
Sep 28 14:40:30 dist-test-slave-06lh krb5kdc[28888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.23.253@KRBTEST.COM
I20260928 14:40:30.344682 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.332957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.6.23.252:37353)
0928 14:40:30.333220 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.333229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.333280 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:40:30.333410 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.333413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.333452 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.333965 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.334085 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.336241 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.337290 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.337630 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.338063 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.340126 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.340130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.340133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.340157 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.341885 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.342685 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.342689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.342690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.342774 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.343082 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.343085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.343087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.343362 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.344339 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.344414 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.344528 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":138,"threads_started":1}
I20260928 14:40:30.345042 29120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5: Bootstrap starting.
I20260928 14:40:30.345722 28969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6: No bootstrap required, opened a new log
I20260928 14:40:30.347797 29044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6: Bootstrap starting.
I20260928 14:40:30.348788 29120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:30.349695 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.328882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.6.23.253:33727)
0928 14:40:30.329005 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.329010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.329095 (+ 85us) server_negotiation.cc:407] Connection header received
0928 14:40:30.329272 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.329275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.329319 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.329410 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.331187 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.332465 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.333770 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.334147 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.334230 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.345603 (+ 11373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.345607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.345609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.345634 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.347434 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.347973 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.347977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.347979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.348066 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.348457 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.348461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.348463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.348717 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.348872 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.349364 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.349495 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:40:30.349712 29120 log.cc:824] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:30.350440 28969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } }
I20260928 14:40:30.350746 28969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6761660fc05e4be7abbeb01c373b01f6, State: Initialized, Role: FOLLOWER
I20260928 14:40:30.351070 29044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6: Neither blocks nor log segments found. Creating new log.
I20260928 14:40:30.351428 28969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } }
I20260928 14:40:30.352069 29044 log.cc:824] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6: Log is configured to *not* fsync() on all Append() calls
I20260928 14:40:30.352196 29130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.352360 29130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:30.353062 28969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:30.354588 29044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6: No bootstrap required, opened a new log
I20260928 14:40:30.354830 29120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5: No bootstrap required, opened a new log
I20260928 14:40:30.355422 6239 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 14:40:30.359105 29142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:30.359192 29142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:30.359025 29044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } }
I20260928 14:40:30.359357 29044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d115b3f905c4323a31c45bc210d60d6, State: Initialized, Role: FOLLOWER
I20260928 14:40:30.359716 29044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } }
I20260928 14:40:30.359701 29120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } }
I20260928 14:40:30.359930 29120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5760a1c4d62c4fa7b84d2aa82a733ec5, State: Initialized, Role: FOLLOWER
I20260928 14:40:30.360240 29120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } }
I20260928 14:40:30.361063 29044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:30.361505 29120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:40:30.359418 28901 master_runner.cc:186] Error getting master registration for 127.6.23.253:33727: OK, instance_id { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" instance_seqno: 1790606430073476 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:40:30.363097 28901 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:40:30.364207 28943 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6761660fc05e4be7abbeb01c373b01f6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56786
I20260928 14:40:30.364460 28943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:40:30.364580 28943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:30.365767 28943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } }
I20260928 14:40:30.366775 28943 leader_election.cc:288] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [CANDIDATE]: Term 1 election: Requested vote from peers 2d115b3f905c4323a31c45bc210d60d6 (127.6.23.253:33727), 5760a1c4d62c4fa7b84d2aa82a733ec5 (127.6.23.252:37353)
I20260928 14:40:30.367477 29145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.367627 29145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:30.367667 29094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6761660fc05e4be7abbeb01c373b01f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5"
W20260928 14:40:30.367820 29166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:30.367889 29166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:30.367923 29094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:30.369308 29094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6761660fc05e4be7abbeb01c373b01f6 in term 1.
I20260928 14:40:30.370038 29018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6761660fc05e4be7abbeb01c373b01f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2d115b3f905c4323a31c45bc210d60d6"
I20260928 14:40:30.370292 29018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:40:30.371729 29018 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6761660fc05e4be7abbeb01c373b01f6 in term 1.
W20260928 14:40:30.371888 29167 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:40:30.371974 29167 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:40:30.374078 28920 leader_election.cc:302] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d115b3f905c4323a31c45bc210d60d6, 6761660fc05e4be7abbeb01c373b01f6; no voters:
I20260928 14:40:30.374643 29130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:40:30.374919 29130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [term 1 LEADER]: Becoming Leader. State: Replica: 6761660fc05e4be7abbeb01c373b01f6, State: Running, Role: LEADER
I20260928 14:40:30.375303 29130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } }
I20260928 14:40:30.377300 29130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6761660fc05e4be7abbeb01c373b01f6. Latest consensus state: current_term: 1 leader_uuid: "6761660fc05e4be7abbeb01c373b01f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.377416 29130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:30.385037 29172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:40:30.386075 29172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:40:30.391656 29144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.391812 29144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:30.393185 28976 master_runner.cc:263] No leader master found from master 2d115b3f905c4323a31c45bc210d60d6
I20260928 14:40:30.393260 28976 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:40:30.394132 29094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [term 1 FOLLOWER]: Refusing update from remote peer 6761660fc05e4be7abbeb01c373b01f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:30.394833 29130 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:30.398845 29145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6761660fc05e4be7abbeb01c373b01f6. Latest consensus state: current_term: 1 leader_uuid: "6761660fc05e4be7abbeb01c373b01f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.398985 29145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:30.400089 29018 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [term 1 FOLLOWER]: Refusing update from remote peer 6761660fc05e4be7abbeb01c373b01f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:40:30.400681 29130 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:40:30.404934 29144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6761660fc05e4be7abbeb01c373b01f6. Latest consensus state: current_term: 1 leader_uuid: "6761660fc05e4be7abbeb01c373b01f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.405084 29144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:30.412324 29170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6761660fc05e4be7abbeb01c373b01f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.412504 29170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:30.412858 29177 mvcc.cc:205] Tried to move back new op lower bound from 7334323938879078400 to 7334323938820919296. Current Snapshot: MvccSnapshot[applied={T|T < 7334323938879078400}]
I20260928 14:40:30.413347 29145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6761660fc05e4be7abbeb01c373b01f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.413486 29145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:30.413916 29181 mvcc.cc:205] Tried to move back new op lower bound from 7334323938879078400 to 7334323938820919296. Current Snapshot: MvccSnapshot[applied={T|T < 7334323938879078400}]
I20260928 14:40:30.415966 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.388698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.6.23.254:35703)
0928 14:40:30.388830 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.388835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.399719 (+ 10884us) server_negotiation.cc:407] Connection header received
0928 14:40:30.401096 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.401100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.401151 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.401202 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.404394 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.405117 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.407276 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.407683 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.407758 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.410421 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.410425 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.410428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.410461 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.412250 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.414509 (+ 2259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.414513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.414515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.414614 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.414920 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.414924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.414926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.415151 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.415294 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.415679 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.415816 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:40:30.419459 29130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6761660fc05e4be7abbeb01c373b01f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.419603 29130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6 [sys.catalog]: This master's current role is: LEADER
I20260928 14:40:30.419893 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.390097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.6.23.253:33727)
0928 14:40:30.390207 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.390212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.394267 (+ 4055us) server_negotiation.cc:407] Connection header received
0928 14:40:30.399804 (+ 5537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.399808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.399871 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.399924 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.402421 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.403218 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.406233 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.406585 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.406665 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.414847 (+ 8182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.414852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.414855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.414901 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.416671 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.418332 (+ 1661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.418336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.418338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.418430 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.418715 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.418718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.418720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.418950 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.419050 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.419568 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.419731 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:40:30.421320 29144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6761660fc05e4be7abbeb01c373b01f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6761660fc05e4be7abbeb01c373b01f6" member_type: VOTER last_known_addr { host: "127.6.23.254" port: 35703 } } peers { permanent_uuid: "2d115b3f905c4323a31c45bc210d60d6" member_type: VOTER last_known_addr { host: "127.6.23.253" port: 33727 } } peers { permanent_uuid: "5760a1c4d62c4fa7b84d2aa82a733ec5" member_type: VOTER last_known_addr { host: "127.6.23.252" port: 37353 } } }
I20260928 14:40:30.421382 29183 mvcc.cc:205] Tried to move back new op lower bound from 7334323938879078400 to 7334323938820919296. Current Snapshot: MvccSnapshot[applied={T|T < 7334323938879078400}]
I20260928 14:40:30.421439 29144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:40:30.423161 29172 catalog_manager.cc:1347] Generated new cluster ID: 165353fe8dbc4b53952f3b6f250468bf
I20260928 14:40:30.423257 29172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:40:30.426044 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:30.389526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.6.23.252:37353)
0928 14:40:30.389652 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.389656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.403293 (+ 13637us) server_negotiation.cc:407] Connection header received
0928 14:40:30.403344 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.403347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.403401 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.403480 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.406184 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.406997 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.413458 (+ 6461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:30.413866 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:30.413946 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:30.421645 (+ 7699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:40:30.421651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:40:30.421654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:40:30.421698 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:40:30.423584 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.424602 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.424607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.424609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.424715 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:40:30.425010 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:40:30.425014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:40:30.425015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:40:30.425243 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:40:30.425362 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:30.425740 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:40:30.425873 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
I20260928 14:40:30.438679 29172 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:40:30.440073 29172 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:40:30.458766 29172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6761660fc05e4be7abbeb01c373b01f6: Generated new TSK 0
I20260928 14:40:30.459860 29172 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:40:30.489500 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.486077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.6.23.253:33727)
0928 14:40:30.486227 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.486232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.488316 (+ 2084us) server_negotiation.cc:407] Connection header received
0928 14:40:30.488523 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.488526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.488579 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.488634 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.489354 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59912: BlockingRecv error: recv got EOF from 127.0.0.1:59912 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 14:40:30.490958 29127 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.486884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.6.23.252:37353)
0928 14:40:30.487004 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.487008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.490022 (+ 3014us) server_negotiation.cc:407] Connection header received
0928 14:40:30.490199 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.490203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.490254 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.490305 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.490832 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40490: BlockingRecv error: recv got EOF from 127.0.0.1:40490 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:40:30.492462 29125 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.487728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.6.23.254:35703)
0928 14:40:30.487842 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.487847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.491551 (+ 3704us) server_negotiation.cc:407] Connection header received
0928 14:40:30.491723 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.491726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.491777 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.491841 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.492342 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56808: BlockingRecv error: recv got EOF from 127.0.0.1:56808 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 14:40:30.501531 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.499743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.6.23.253:33727)
0928 14:40:30.499877 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.499882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.500439 (+ 557us) server_negotiation.cc:407] Connection header received
0928 14:40:30.500684 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.500688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.500770 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.500838 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.501410 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59922: BlockingRecv error: recv got EOF from 127.0.0.1:59922 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 14:40:30.506920 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.505276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.6.23.253:33727)
0928 14:40:30.505386 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.505390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.505957 (+ 567us) server_negotiation.cc:407] Connection header received
0928 14:40:30.506150 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.506154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.506205 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.506258 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.506779 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59934: BlockingRecv error: recv got EOF from 127.0.0.1:59934 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 14:40:30.512951 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.511189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.6.23.253:33727)
0928 14:40:30.511377 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.511382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.511967 (+ 585us) server_negotiation.cc:407] Connection header received
0928 14:40:30.512168 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.512172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.512227 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.512280 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.512831 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59938: BlockingRecv error: recv got EOF from 127.0.0.1:59938 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260928 14:40:30.521032 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.519222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.6.23.253:33727)
0928 14:40:30.519405 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.519411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.520030 (+ 619us) server_negotiation.cc:407] Connection header received
0928 14:40:30.520238 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.520242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.520296 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.520350 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.520908 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59944: BlockingRecv error: recv got EOF from 127.0.0.1:59944 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 14:40:30.533550 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.531620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59948 (local address 127.6.23.253:33727)
0928 14:40:30.531772 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.531776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.532500 (+ 724us) server_negotiation.cc:407] Connection header received
0928 14:40:30.532728 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.532733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.532791 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.532846 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.533422 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59948: BlockingRecv error: recv got EOF from 127.0.0.1:59948 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 14:40:30.554478 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.552304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59964 (local address 127.6.23.253:33727)
0928 14:40:30.552441 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.552446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.553183 (+ 737us) server_negotiation.cc:407] Connection header received
0928 14:40:30.553433 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.553439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.553515 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.553585 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.554347 (+ 762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59964: BlockingRecv error: recv got EOF from 127.0.0.1:59964 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 14:40:30.590740 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.589292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59966 (local address 127.6.23.253:33727)
0928 14:40:30.589418 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.589423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.589744 (+ 321us) server_negotiation.cc:407] Connection header received
0928 14:40:30.589901 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.589904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.589963 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.590069 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.590615 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59966: BlockingRecv error: recv got EOF from 127.0.0.1:59966 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:40:30.658098 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.656727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59974 (local address 127.6.23.253:33727)
0928 14:40:30.656940 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.656945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.657106 (+ 161us) server_negotiation.cc:407] Connection header received
0928 14:40:30.657299 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.657303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.657362 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.657453 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.657947 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59974: BlockingRecv error: recv got EOF from 127.0.0.1:59974 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260928 14:40:30.789582 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:30.788543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.6.23.253:33727)
0928 14:40:30.788718 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:30.788722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:30.788808 (+ 86us) server_negotiation.cc:407] Connection header received
0928 14:40:30.788931 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:30.788934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:30.788984 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:30.789071 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:30.789484 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59990: BlockingRecv error: recv got EOF from 127.0.0.1:59990 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 14:40:31.049400 29049 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:40:31.048111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59992 (local address 127.6.23.253:33727)
0928 14:40:31.048320 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:31.048325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:31.048342 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:40:31.048415 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:31.048419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:31.048626 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:31.048737 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:40:31.049231 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59992: BlockingRecv error: recv got EOF from 127.0.0.1:59992 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 14:40:31.369243 29166 catalog_manager.cc:1259] Loaded cluster ID: 165353fe8dbc4b53952f3b6f250468bf
I20260928 14:40:31.369360 29166 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5: loading cluster ID for follower catalog manager: success
I20260928 14:40:31.371115 29166 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5: acquiring CA information for follower catalog manager: success
I20260928 14:40:31.372571 29166 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5760a1c4d62c4fa7b84d2aa82a733ec5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:40:31.373235 29167 catalog_manager.cc:1259] Loaded cluster ID: 165353fe8dbc4b53952f3b6f250468bf
I20260928 14:40:31.373308 29167 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6: loading cluster ID for follower catalog manager: success
I20260928 14:40:31.375056 29167 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6: acquiring CA information for follower catalog manager: success
I20260928 14:40:31.376256 29167 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2d115b3f905c4323a31c45bc210d60d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:40:31.567747 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:31.563835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.6.23.253:33727)
0928 14:40:31.564062 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:31.564065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:31.564082 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:40:31.564205 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:31.564208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:31.564355 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:31.564471 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:31.564784 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.565338 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.566268 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.566501 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.566585 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:31.566905 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:31.567416 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:31.567508 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:31.567569 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 14:40:31.567618 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260928 14:40:31.575380 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:31.571629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40492 (local address 127.6.23.252:37353)
0928 14:40:31.571831 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:31.571834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:31.571846 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:40:31.571924 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:31.571926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:31.572053 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:31.572208 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:31.572438 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.572989 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.573882 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.574096 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.574215 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:31.574522 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:31.574950 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:31.575089 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:31.575132 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 14:40:31.575197 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":61,"threads_started":1}
I20260928 14:40:31.582597 29253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:31.578410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60006 (local address 127.6.23.253:33727)
0928 14:40:31.578567 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:31.578573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:31.578777 (+ 204us) server_negotiation.cc:407] Connection header received
0928 14:40:31.578993 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:31.578996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:31.579058 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:31.579139 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:31.579422 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.580242 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.581423 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.581674 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.581787 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:31.582106 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:31.582210 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:31.582298 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:31.582387 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 14:40:31.582441 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:40:31.583284 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:31.578896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40502 (local address 127.6.23.252:37353)
0928 14:40:31.579022 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:31.579026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:31.579622 (+ 596us) server_negotiation.cc:407] Connection header received
0928 14:40:31.579730 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:31.579733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:31.579777 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:31.579864 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:31.580086 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.580786 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.582015 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.582337 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.582411 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:31.582789 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:31.582860 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:31.582906 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:31.583061 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 14:40:31.583105 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:40:31.583599 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:40:31.578351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.6.23.254:35703)
0928 14:40:31.578713 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 14:40:31.578719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:40:31.578738 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:40:31.578899 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:40:31.578905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:40:31.579065 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:40:31.579190 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:40:31.579532 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.580319 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.581492 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:40:31.581856 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:40:31.581973 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:40:31.582249 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:40:31.582604 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:40:31.582687 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:40:31.583379 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 14:40:31.583447 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":95,"threads_started":1}
I20260928 14:40:31.586591 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28901
I20260928 14:40:31.593410 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 28976
I20260928 14:40:31.599520 6239 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskpOhn9p/build/debug/bin/kudu with pid 29051
2026-09-28T14:40:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3112 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6260 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288785 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST